2024-03-26T04:28:13.115 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T04:28:13.115 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029 --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --description orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} --verbose -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/orig.config.yaml 2024-03-26T04:28:13.141 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:28:13.214 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7622029' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGiafEtzcNaIpyCQD5qdE71/Oo+/biy6vk9ctDr7hWRANeqskraL43haMAU048ObshHA6lh5GDuWWa8J9fngZg= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITubfsMWWTDhcw8ayx3uCTttBhyLSzVmOig8oejOE/sjV+55swSQI7hWPvPptVuaICc2+CZcmlD32vjm4LCcLk= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T04:28:13.214 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T04:28:13.215 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T04:28:13.215 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T04:28:13.217 INFO:teuthology.task.internal:Checking packages... 2024-03-26T04:28:13.240 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T04:28:13.240 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T04:28:13.240 INFO:teuthology.packaging:ref: None 2024-03-26T04:28:13.240 INFO:teuthology.packaging:tag: None 2024-03-26T04:28:13.240 INFO:teuthology.packaging:branch: squid 2024-03-26T04:28:13.240 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:28:13.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T04:28:13.396 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T04:28:13.397 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T04:28:13.404 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T04:28:13.404 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T04:28:13.447 INFO:teuthology.task.internal:Saving configuration 2024-03-26T04:28:13.458 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T04:28:13.472 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:28:13.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.726972', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGiafEtzcNaIpyCQD5qdE71/Oo+/biy6vk9ctDr7hWRANeqskraL43haMAU048ObshHA6lh5GDuWWa8J9fngZg='} 2024-03-26T04:28:13.515 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.728498', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITubfsMWWTDhcw8ayx3uCTttBhyLSzVmOig8oejOE/sjV+55swSQI7hWPvPptVuaICc2+CZcmlD32vjm4LCcLk='} 2024-03-26T04:28:13.515 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T04:28:13.521 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T04:28:13.522 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T04:28:13.522 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T04:28:13.617 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4cbaed8f70>, signals=[15]) 2024-03-26T04:28:13.617 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T04:28:13.624 INFO:teuthology.task.internal:Opening connections... 2024-03-26T04:28:13.624 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-03-26T04:28:13.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:28:13.730 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-03-26T04:28:13.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:28:13.821 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T04:28:13.826 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-03-26T04:28:13.843 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-03-26T04:28:13.844 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-03-26T04:28:13.900 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-03-26T04:28:13.900 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:28:13.901 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:28:13.902 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-03-26T04:28:13.926 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-03-26T04:28:13.944 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-03-26T04:28:13.944 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:28:14.001 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:28:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:28:14.002 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-03-26T04:28:14.025 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T04:28:14.033 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T04:28:14.055 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T04:28:14.055 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:28:14.057 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:28:14.073 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T04:28:14.079 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T04:28:14.079 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:28:14.115 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:28:14.217 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T04:28:14.268 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T04:28:14.285 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T04:28:14.285 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T04:28:14.285 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:28:14.286 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:28:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:28:14.302 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-26T04:28:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:28:14.303 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-26T04:28:14.359 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:28:14.360 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-03-26T04:28:14.360 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-03-26T04:28:14.361 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:28:14.361 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-361.el9.x86_64 2024-03-26T04:28:14.361 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2024-03-26T04:28:16.576 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T04:28:16.580 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T04:28:18.014 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-26T04:28:18.035 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-26T04:28:18.809 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-26T04:28:20.877 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-26T04:28:20.938 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-03-26T04:28:23.216 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-26T04:28:24.902 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-26T04:28:26.528 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-26T04:28:27.840 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-26T04:28:28.315 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-26T04:28:30.149 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-26T04:28:33.004 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-26T04:28:35.269 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:28:35.269 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:28:35.359 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:28:35.368 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2024-03-26T04:28:35.369 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:28:35.370 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 76 M 2024-03-26T04:28:35.370 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 123 M 2024-03-26T04:28:35.370 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:28:35.822 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 348 kB/s | 66 kB 00:00 2024-03-26T04:28:39.621 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.7 MB/s | 15 MB 00:03 2024-03-26T04:28:42.188 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 4.3 MB/s | 27 MB 00:06 2024-03-26T04:28:43.022 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 4.5 MB/s | 33 MB 00:07 2024-03-26T04:28:43.022 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:28:43.022 INFO:teuthology.orchestra.run.smithi012.stdout:Total 9.9 MB/s | 76 MB 00:07 2024-03-26T04:28:43.714 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:28:44.157 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:28:44.157 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:28:45.211 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:28:45.211 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:28:45.832 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 21 MB 00:15 2024-03-26T04:28:47.810 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:28:48.429 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:28:48.521 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:50.707 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-26T04:28:51.371 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:28:51.508 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:52.863 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:28:52.864 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:28:52.951 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:28:52.958 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:28:52.958 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-26T04:28:52.958 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-03-26T04:28:52.959 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:28:52.960 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 76 M 2024-03-26T04:28:52.960 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-03-26T04:28:52.960 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:28:53.201 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.202 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.202 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.427 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.445 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.446 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.641 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 118 kB/s | 66 kB 00:00 2024-03-26T04:28:53.709 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:53.777 INFO:teuthology.orchestra.run.smithi186.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:28:56.442 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.4 MB/s | 15 MB 00:03 2024-03-26T04:28:56.579 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:28:56.617 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:28:58.817 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.3 MB/s | 27 MB 00:05 2024-03-26T04:29:00.760 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 4.4 MB/s | 33 MB 00:07 2024-03-26T04:29:00.763 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:29:00.763 INFO:teuthology.orchestra.run.smithi186.stdout:Total 9.7 MB/s | 76 MB 00:07 2024-03-26T04:29:01.495 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:29:01.712 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:01.884 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:29:01.885 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:29:02.882 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:29:02.883 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:29:05.367 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:29:05.991 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:29:06.091 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:29:08.826 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:29:08.960 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:29:13.697 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:29:14.019 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:18.713 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:32.382 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:32.383 INFO:teuthology.orchestra.run.smithi012.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:29:32.383 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:29:32.429 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:32.791 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:32.791 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:29:32.791 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:29:32.791 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:29:33.269 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:33.269 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:29:33.269 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:29:33.269 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:33.270 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:29:33.270 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:29:33.270 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:29:33.270 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:29:33.270 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:29:33.574 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:29:33.667 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:33.667 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:29:33.667 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:29:33.667 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:29:33.667 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:29:33.667 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T04:29:33.668 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:29:33.668 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:29:33.751 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:33.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:29:33.751 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2024-03-26T04:29:33.789 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:29:33.790 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:29:33.791 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:29:33.792 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:29:33.792 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:29:33.792 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:29:34.340 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2024-03-26T04:29:35.770 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:29:35.797 INFO:teuthology.orchestra.run.smithi012.stderr:done 2024-03-26T04:29:35.800 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2024-03-26T04:29:35.815 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.1EQ9PioJsV 2024-03-26T04:29:35.815 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1EQ9PioJsV 2024-03-26T04:29:35.887 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.1EQ9PioJsV 2024-03-26T04:29:36.029 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.1EQ9PioJsV is 6KB 2024-03-26T04:29:36.079 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.1EQ9PioJsV 2024-03-26T04:29:36.093 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:29:36.170 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:29:36.170 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:29:36.171 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:29:36.171 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:29:36.172 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:29:36.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2024-03-26T04:29:47.908 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:47.909 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:29:47.909 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:29:47.953 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:48.338 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:48.338 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:29:48.338 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:29:48.338 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:29:48.769 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:29:49.101 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:29:49.205 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:49.205 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:29:49.206 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:29:49.206 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:29:49.206 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:29:49.206 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T04:29:49.206 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:29:49.206 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:29:49.286 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:29:49.286 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:29:49.287 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:29:49.327 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:29:49.328 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:29:49.329 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:29:49.329 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:29:49.329 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:29:49.896 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-03-26T04:29:51.270 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:29:51.309 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-03-26T04:29:51.311 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-26T04:29:51.326 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.HJPuGsfNTD 2024-03-26T04:29:51.326 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HJPuGsfNTD 2024-03-26T04:29:51.394 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.HJPuGsfNTD 2024-03-26T04:29:51.537 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.HJPuGsfNTD is 6KB 2024-03-26T04:29:51.585 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.HJPuGsfNTD 2024-03-26T04:29:51.598 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:29:51.672 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:29:51.672 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:29:51.672 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:29:51.672 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:29:51.674 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:29:51.832 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-03-26T04:30:06.354 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T04:30:06.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-26T04:30:06.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:21.850 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T04:30:21.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:30:21.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:24.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-26T04:30:33.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-26T04:30:33.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:40.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-26T04:30:49.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:30:49.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:30:52.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2024-03-26T04:30:52.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-26T04:31:04.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-26T04:31:04.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:31:04.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:31:04.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:31:06.223 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-26T04:31:06.847 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-26T04:31:06.847 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:31:06.847 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-03-26T04:31:06.863 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:31:06.863 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:31:06.863 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:31:06.863 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:31:07.864 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T04:31:07.864 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:31:07.992 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2024-03-26T04:31:08.044 DEBUG:teuthology.parallel:result is None 2024-03-26T04:32:04.483 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T04:32:19.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:32:19.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:32:19.778 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-26T04:32:20.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:32:20.149 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:32:20.150 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-26T04:32:20.166 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:32:20.166 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:32:20.166 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:32:20.166 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:32:21.167 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T04:32:21.167 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:32:21.230 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-03-26T04:32:21.265 DEBUG:teuthology.parallel:result is None 2024-03-26T04:32:21.265 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T04:32:21.272 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T04:32:21.273 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:32:21.276 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:32:21.293 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T04:32:21.312 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T04:32:21.387 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T04:32:21.388 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:32:21.390 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:32:21.424 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T04:32:21.454 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T04:32:21.455 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:32:21.457 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:32:21.497 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:32:21.499 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:32:21.511 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:32:21.513 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:32:21.515 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T04:32:21.537 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T04:32:21.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:32:21.556 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:32:21.586 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T04:32:21.616 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T04:32:21.616 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:32:21.625 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:32:21.641 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:32:21.721 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:32:21.773 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:32:21.853 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:32:21.897 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:32:21.897 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:32:21.963 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:32:22.007 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:32:22.067 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:32:22.147 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:32:22.195 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:32:22.195 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:32:22.264 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-03-26T04:32:22.266 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-03-26T04:32:22.311 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:32:22.350 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:32:22.748 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T04:32:22.757 INFO:teuthology.task.internal:Starting timer... 2024-03-26T04:32:22.758 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T04:32:22.807 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T04:32:22.865 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T04:32:22.865 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-03-26T04:32:22.934 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-03-26T04:32:22.937 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-03-26T04:32:22.992 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-03-26T04:32:22.994 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T04:32:22.994 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-03-26T04:32:23.015 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-03-26T04:32:23.015 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-03-26T04:32:23.036 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-03-26T04:32:23.036 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-03-26T04:32:23.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:32:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:32:23.083 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:32:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:32:23.112 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T04:32:23.112 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:32:23.159 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:32:23.196 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T04:32:23.207 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T04:32:23.223 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T04:32:23.224 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T04:36:40.487 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-03-26T04:36:40.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-26T04:36:40.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:36:40.580 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-03-26T04:36:40.664 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-03-26T04:36:40.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:36:40.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:36:40.755 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-26T04:36:40.832 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-26T04:36:40.833 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T04:36:40.844 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T04:36:40.844 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:36:40.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:36:40.847 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:36:40.847 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:36:40.875 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:36:40.893 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:36:40.903 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:36:40.919 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:36:40.922 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-03-26T04:36:40.935 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-03-26T04:36:40.949 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-03-26T04:36:40.952 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:36:40.962 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-03-26T04:36:40.968 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:36:40.978 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:36:40.995 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:36:41.019 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:36:41.022 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:36:41.022 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-26T04:36:41.022 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.023 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.023 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.023 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.046 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:36:41.050 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:36:41.050 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-26T04:36:41.050 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.051 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.051 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.051 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:36:41.051 INFO:teuthology.run_tasks:Running task install... 2024-03-26T04:36:41.061 DEBUG:teuthology.task.install:project ceph 2024-03-26T04:36:41.062 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}} 2024-03-26T04:36:41.062 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T04:36:41.062 INFO:teuthology.task.install:Using flavor: default 2024-03-26T04:36:41.068 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T04:36:41.068 INFO:teuthology.task.install:extra packages: [] 2024-03-26T04:36:41.068 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T04:36:41.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:36:41.071 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T04:36:41.071 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:36:41.214 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T04:36:41.214 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T04:36:41.309 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T04:36:41.309 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T04:36:41.416 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T04:36:41.416 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:36:41.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T04:36:41.418 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T04:36:41.419 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:36:41.419 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T04:36:41.447 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T04:36:41.447 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T04:36:41.448 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T04:36:41.448 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T04:36:41.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T04:36:41.516 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T04:36:41.595 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T04:36:41.596 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T04:36:41.661 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-03-26T04:36:41.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-03-26T04:36:41.663 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-03-26T04:36:41.665 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-03-26T04:36:41.905 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-03-26T04:36:41.916 INFO:teuthology.orchestra.run.smithi012.stdout:74 files removed 2024-03-26T04:36:41.932 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw 2024-03-26T04:36:41.943 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw 2024-03-26T04:36:42.544 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 265 kB/s | 83 kB 00:00 2024-03-26T04:36:42.758 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 163 kB/s | 83 kB 00:00 2024-03-26T04:36:42.987 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-26T04:36:43.125 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-26T04:36:43.420 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-26T04:36:43.610 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-03-26T04:36:44.181 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.7 MB 00:00 2024-03-26T04:36:44.353 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.7 MB 00:00 2024-03-26T04:36:45.573 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-26T04:36:45.820 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-26T04:36:46.686 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T04:36:48.495 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-26T04:36:49.800 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-26T04:36:51.632 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-26T04:36:53.204 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-26T04:36:54.688 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-26T04:36:56.193 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T04:36:56.975 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-26T04:36:58.783 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-26T04:37:00.343 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-26T04:37:01.431 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-26T04:37:04.028 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T04:37:04.038 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T04:37:04.039 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T04:37:04.040 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:Install 30 Packages 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 2 Packages 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 58 M 2024-03-26T04:37:04.041 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:37:05.310 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-26T04:37:05.399 INFO:teuthology.orchestra.run.smithi012.stdout:(1/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9. 8.6 MB/s | 9.8 MB 00:01 2024-03-26T04:37:05.449 INFO:teuthology.orchestra.run.smithi012.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 505 kB/s | 25 kB 00:00 2024-03-26T04:37:05.549 INFO:teuthology.orchestra.run.smithi012.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 7.3 MB/s | 744 kB 00:00 2024-03-26T04:37:05.641 INFO:teuthology.orchestra.run.smithi012.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 5.3 MB/s | 498 kB 00:00 2024-03-26T04:37:05.833 INFO:teuthology.orchestra.run.smithi012.stdout:(5/32): ceph-common-19.0.0-1775.g5e70124b.el9.x 13 MB/s | 20 MB 00:01 2024-03-26T04:37:05.875 INFO:teuthology.orchestra.run.smithi012.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 1.1 MB/s | 45 kB 00:00 2024-03-26T04:37:05.959 INFO:teuthology.orchestra.run.smithi012.stdout:(7/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-26T04:37:05.984 INFO:teuthology.orchestra.run.smithi012.stdout:(8/32): python3-ceph-common-19.0.0-1775.g5e7012 1.2 MB/s | 131 kB 00:00 2024-03-26T04:37:06.009 INFO:teuthology.orchestra.run.smithi012.stdout:(9/32): python3-cephfs-19.0.0-1775.g5e70124b.el 3.2 MB/s | 162 kB 00:00 2024-03-26T04:37:06.034 INFO:teuthology.orchestra.run.smithi012.stdout:(10/32): python3-rados-19.0.0-1775.g5e70124b.el 6.2 MB/s | 321 kB 00:00 2024-03-26T04:37:06.060 INFO:teuthology.orchestra.run.smithi012.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 5.8 MB/s | 298 kB 00:00 2024-03-26T04:37:06.085 INFO:teuthology.orchestra.run.smithi012.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T04:37:06.269 INFO:teuthology.orchestra.run.smithi012.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 508 kB/s | 106 kB 00:00 2024-03-26T04:37:06.294 INFO:teuthology.orchestra.run.smithi012.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 877 kB/s | 159 kB 00:00 2024-03-26T04:37:06.319 INFO:teuthology.orchestra.run.smithi012.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-26T04:37:06.389 INFO:teuthology.orchestra.run.smithi012.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-26T04:37:06.411 INFO:teuthology.orchestra.run.smithi012.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-26T04:37:06.437 INFO:teuthology.orchestra.run.smithi012.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-03-26T04:37:06.503 INFO:teuthology.orchestra.run.smithi012.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:37:06.503 INFO:teuthology.orchestra.run.smithi012.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:37:06.669 INFO:teuthology.orchestra.run.smithi012.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:37:06.698 INFO:teuthology.orchestra.run.smithi012.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:37:07.020 INFO:teuthology.orchestra.run.smithi012.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 61 kB/s | 36 kB 00:00 2024-03-26T04:37:07.057 INFO:teuthology.orchestra.run.smithi012.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:37:07.090 INFO:teuthology.orchestra.run.smithi012.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:37:07.237 INFO:teuthology.orchestra.run.smithi012.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 267 kB/s | 221 kB 00:00 2024-03-26T04:37:07.288 INFO:teuthology.orchestra.run.smithi012.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 177 kB/s | 47 kB 00:00 2024-03-26T04:37:07.463 INFO:teuthology.orchestra.run.smithi012.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-26T04:37:07.505 INFO:teuthology.orchestra.run.smithi012.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 632 kB/s | 26 kB 00:00 2024-03-26T04:37:07.547 INFO:teuthology.orchestra.run.smithi012.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-26T04:37:07.589 INFO:teuthology.orchestra.run.smithi012.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-26T04:37:07.689 INFO:teuthology.orchestra.run.smithi012.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.3 MB/s | 838 kB 00:00 2024-03-26T04:37:07.746 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:37:07.748 INFO:teuthology.orchestra.run.smithi012.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options 2024-03-26T04:37:07.762 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T04:37:07.763 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T04:37:07.764 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout:Install 30 Packages 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 2 Packages 2024-03-26T04:37:07.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:07.766 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 58 M 2024-03-26T04:37:07.766 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:37:07.773 INFO:teuthology.orchestra.run.smithi012.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-03-26T04:37:07.840 INFO:teuthology.orchestra.run.smithi012.stdout:(29/32): ceph-base-19.0.0-1775.g5e70124b.el9.x8 1.5 MB/s | 5.3 MB 00:03 2024-03-26T04:37:07.873 INFO:teuthology.orchestra.run.smithi012.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-26T04:37:08.099 INFO:teuthology.orchestra.run.smithi012.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-26T04:37:09.211 INFO:teuthology.orchestra.run.smithi186.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 4.6 MB/s | 5.3 MB 00:01 2024-03-26T04:37:09.274 INFO:teuthology.orchestra.run.smithi012.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 2.2 MB/s | 3.1 MB 00:01 2024-03-26T04:37:09.281 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:37:09.281 INFO:teuthology.orchestra.run.smithi012.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-26T04:37:09.317 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:37:09.320 INFO:teuthology.orchestra.run.smithi186.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 232 kB/s | 25 kB 00:00 2024-03-26T04:37:09.359 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:37:09.359 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:37:09.478 INFO:teuthology.orchestra.run.smithi186.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 4.6 MB/s | 744 kB 00:00 2024-03-26T04:37:09.578 INFO:teuthology.orchestra.run.smithi186.stdout:(4/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9. 6.4 MB/s | 9.8 MB 00:01 2024-03-26T04:37:09.604 INFO:teuthology.orchestra.run.smithi186.stdout:(5/32): libradosstriper1-19.0.0-1775.g5e70124b. 3.9 MB/s | 498 kB 00:00 2024-03-26T04:37:09.704 INFO:teuthology.orchestra.run.smithi186.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 451 kB/s | 45 kB 00:00 2024-03-26T04:37:09.796 INFO:teuthology.orchestra.run.smithi186.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 1.4 MB/s | 131 kB 00:00 2024-03-26T04:37:09.884 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:37:09.884 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:37:09.913 INFO:teuthology.orchestra.run.smithi186.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 1.4 MB/s | 162 kB 00:00 2024-03-26T04:37:10.030 INFO:teuthology.orchestra.run.smithi186.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 2.7 MB/s | 321 kB 00:00 2024-03-26T04:37:10.097 INFO:teuthology.orchestra.run.smithi186.stdout:(10/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_ 9.9 MB/s | 5.1 MB 00:00 2024-03-26T04:37:10.122 INFO:teuthology.orchestra.run.smithi186.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 3.2 MB/s | 298 kB 00:00 2024-03-26T04:37:10.181 INFO:teuthology.orchestra.run.smithi186.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.2 MB/s | 99 kB 00:00 2024-03-26T04:37:10.239 INFO:teuthology.orchestra.run.smithi186.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 905 kB/s | 106 kB 00:00 2024-03-26T04:37:10.265 INFO:teuthology.orchestra.run.smithi186.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-26T04:37:10.290 INFO:teuthology.orchestra.run.smithi186.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-26T04:37:10.349 INFO:teuthology.orchestra.run.smithi186.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-26T04:37:10.374 INFO:teuthology.orchestra.run.smithi186.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-26T04:37:10.407 INFO:teuthology.orchestra.run.smithi186.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 705 kB/s | 41 kB 00:00 2024-03-26T04:37:10.691 INFO:teuthology.orchestra.run.smithi186.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 127 kB/s | 36 kB 00:00 2024-03-26T04:37:10.908 INFO:teuthology.orchestra.run.smithi186.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 218 kB/s | 47 kB 00:00 2024-03-26T04:37:10.950 INFO:teuthology.orchestra.run.smithi186.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 383 kB/s | 221 kB 00:00 2024-03-26T04:37:11.424 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:37:11.425 INFO:teuthology.orchestra.run.smithi186.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 597 kB/s | 309 kB 00:00 2024-03-26T04:37:11.510 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T04:37:11.545 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T04:37:11.570 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T04:37:11.610 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:37:11.627 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:37:11.684 INFO:teuthology.orchestra.run.smithi186.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-26T04:37:11.819 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:37:11.890 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:37:12.026 INFO:teuthology.orchestra.run.smithi186.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-26T04:37:12.117 INFO:teuthology.orchestra.run.smithi186.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-26T04:37:12.289 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T04:37:12.314 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:37:12.385 INFO:teuthology.orchestra.run.smithi186.stdout:(26/32): ceph-common-19.0.0-1775.g5e70124b.el9. 4.7 MB/s | 20 MB 00:04 2024-03-26T04:37:12.390 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:37:12.410 INFO:teuthology.orchestra.run.smithi186.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 175 kB/s | 67 kB 00:00 2024-03-26T04:37:12.480 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T04:37:12.507 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:37:12.553 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:37:12.582 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T04:37:12.626 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T04:37:12.696 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:37:12.718 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:37:12.765 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:37:12.800 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T04:37:12.893 INFO:teuthology.orchestra.run.smithi186.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-03-26T04:37:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T04:37:12.968 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T04:37:12.969 INFO:teuthology.orchestra.run.smithi186.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 985 kB/s | 838 kB 00:00 2024-03-26T04:37:13.008 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T04:37:13.041 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T04:37:13.070 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T04:37:13.103 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T04:37:13.169 INFO:teuthology.orchestra.run.smithi186.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-26T04:37:13.427 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T04:37:13.466 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:37:13.557 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:37:13.587 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T04:37:13.629 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T04:37:13.661 INFO:teuthology.orchestra.run.smithi186.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 4.4 MB/s | 3.4 MB 00:00 2024-03-26T04:37:14.112 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T04:37:14.135 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:37:14.189 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:37:14.262 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T04:37:14.304 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T04:37:14.870 INFO:teuthology.orchestra.run.smithi186.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 1.6 MB/s | 3.1 MB 00:01 2024-03-26T04:37:14.873 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:37:14.873 INFO:teuthology.orchestra.run.smithi186.stdout:Total 8.2 MB/s | 58 MB 00:07 2024-03-26T04:37:14.909 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:37:14.947 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:37:14.948 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:37:15.431 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:37:15.431 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:37:16.647 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:37:16.692 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:37:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:37:16.957 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T04:37:17.064 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T04:37:17.125 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T04:37:17.156 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:37:17.173 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:37:17.276 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:37:17.313 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:37:17.365 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:37:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:37:17.404 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T04:37:17.404 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T04:37:17.404 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:17.426 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:37:17.431 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:37:17.807 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T04:37:17.851 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:37:17.943 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:37:18.032 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T04:37:18.061 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:37:18.133 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:37:18.170 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T04:37:18.217 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T04:37:18.291 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:37:18.322 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:37:18.390 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:37:18.424 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T04:37:18.560 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T04:37:18.606 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T04:37:18.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T04:37:18.678 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T04:37:18.707 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T04:37:18.739 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T04:37:19.056 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T04:37:19.087 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:37:19.177 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:37:19.208 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T04:37:19.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T04:37:19.715 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T04:37:19.747 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:37:19.818 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:37:19.888 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T04:37:19.916 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T04:37:22.135 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:37:22.221 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:37:22.778 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:37:22.816 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:37:22.906 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:37:22.906 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T04:37:22.906 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T04:37:22.907 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:22.943 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:37:27.508 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:37:27.508 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-03-26T04:37:27.508 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-03-26T04:37:27.508 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-03-26T04:37:27.508 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-03-26T04:37:27.508 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-03-26T04:37:27.509 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-03-26T04:37:27.509 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-03-26T04:37:27.509 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:28.450 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:37:28.474 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:37:28.474 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:28.474 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T04:37:28.474 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:37:28.474 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:37:28.474 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:28.476 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:37:28.561 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:37:28.561 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:37:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:37:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T04:37:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T04:37:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T04:37:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T04:37:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T04:37:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T04:37:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:37:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T04:37:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T04:37:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T04:37:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T04:37:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:30.769 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:37:30.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-test 2024-03-26T04:37:31.559 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:30 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:37:31.720 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:37:31.721 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T04:37:31.722 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 48 M 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 205 M 2024-03-26T04:37:31.723 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:37:31.928 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-26T04:37:32.020 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 300 kB 00:00 2024-03-26T04:37:32.070 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-26T04:37:32.179 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 390 kB/s | 163 kB 00:00 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-03-26T04:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-03-26T04:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-03-26T04:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:32.613 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 345 kB/s | 187 kB 00:00 2024-03-26T04:37:33.517 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:37:33.542 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:37:33.542 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:33.542 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T04:37:33.542 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:37:33.543 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:37:33.543 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:33.544 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:37:33.611 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:37:33.612 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:37:34.761 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:37:34.761 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T04:37:34.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T04:37:34.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T04:37:34.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T04:37:34.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T04:37:34.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T04:37:34.765 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:37:34.765 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T04:37:34.765 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T04:37:35.346 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 13 MB/s | 47 MB 00:03 2024-03-26T04:37:35.349 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:37:35.349 INFO:teuthology.orchestra.run.smithi012.stdout:Total 13 MB/s | 48 MB 00:03 2024-03-26T04:37:35.364 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:37:35.386 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:37:35.387 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:37:35.778 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:37:35.778 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.996 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:35.998 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:37:36.343 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test 2024-03-26T04:37:36.366 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:37:36.448 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T04:37:36.489 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T04:37:36.520 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:37:36.563 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T04:37:36.590 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:37:36.895 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:31 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:37:37.053 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:37:37.053 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:37:37.054 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:37:37.055 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-03-26T04:37:37.055 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:37.055 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 48 M 2024-03-26T04:37:37.055 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 205 M 2024-03-26T04:37:37.055 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:37:37.433 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 891 kB/s | 217 kB 00:00 2024-03-26T04:37:37.559 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-26T04:37:37.634 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 834 kB/s | 63 kB 00:00 2024-03-26T04:37:37.734 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 300 kB/s | 163 kB 00:00 2024-03-26T04:37:37.984 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 534 kB/s | 187 kB 00:00 2024-03-26T04:37:39.901 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 17 MB/s | 47 MB 00:02 2024-03-26T04:37:39.904 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:37:39.905 INFO:teuthology.orchestra.run.smithi186.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-26T04:37:39.919 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:37:39.939 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:37:39.939 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:37:40.322 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:37:40.323 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:37:40.928 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:37:41.019 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T04:37:41.055 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T04:37:41.084 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:37:41.126 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T04:37:41.154 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:37:41.315 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:37:41.399 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:37:43.708 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:37:43.708 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T04:37:43.708 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T04:37:43.708 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:37:43.708 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T04:37:43.709 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T04:37:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T04:37:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:44.082 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:37:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:44.082 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2024-03-26T04:37:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T04:37:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T04:37:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T04:37:44.083 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:44.083 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:37:44.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph 2024-03-26T04:37:44.967 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:43 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:37:45.126 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout:========================================================================================= 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout:========================================================================================= 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T04:37:45.129 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T04:37:45.130 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T04:37:45.131 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T04:37:45.132 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout:========================================================================================= 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout:Install 36 Packages 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 30 M 2024-03-26T04:37:45.133 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 107 M 2024-03-26T04:37:45.134 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:37:45.638 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:37:45.721 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:37:46.039 INFO:teuthology.orchestra.run.smithi012.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-26T04:37:46.239 INFO:teuthology.orchestra.run.smithi012.stdout:(2/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 4.1 MB/s | 1.5 MB 00:00 2024-03-26T04:37:46.623 INFO:teuthology.orchestra.run.smithi012.stdout:(3/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86_ 8.0 MB/s | 4.7 MB 00:00 2024-03-26T04:37:46.715 INFO:teuthology.orchestra.run.smithi012.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 2.7 MB/s | 248 kB 00:00 2024-03-26T04:37:46.798 INFO:teuthology.orchestra.run.smithi012.stdout:(5/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 2.5 MB/s | 2.3 MB 00:00 2024-03-26T04:37:46.873 INFO:teuthology.orchestra.run.smithi012.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-26T04:37:46.898 INFO:teuthology.orchestra.run.smithi012.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2024-03-26T04:37:46.925 INFO:teuthology.orchestra.run.smithi012.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-26T04:37:46.949 INFO:teuthology.orchestra.run.smithi012.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-26T04:37:46.974 INFO:teuthology.orchestra.run.smithi012.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-26T04:37:47.000 INFO:teuthology.orchestra.run.smithi012.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T04:37:47.059 INFO:teuthology.orchestra.run.smithi012.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-26T04:37:47.083 INFO:teuthology.orchestra.run.smithi012.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-26T04:37:47.109 INFO:teuthology.orchestra.run.smithi012.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T04:37:47.134 INFO:teuthology.orchestra.run.smithi012.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-26T04:37:47.301 INFO:teuthology.orchestra.run.smithi012.stdout:(16/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86 16 MB/s | 17 MB 00:01 2024-03-26T04:37:47.360 INFO:teuthology.orchestra.run.smithi012.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-26T04:37:47.502 INFO:teuthology.orchestra.run.smithi012.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 318 kB/s | 125 kB 00:00 2024-03-26T04:37:47.913 INFO:teuthology.orchestra.run.smithi012.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 79 kB/s | 43 kB 00:00 2024-03-26T04:37:48.044 INFO:teuthology.orchestra.run.smithi012.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 204 kB/s | 151 kB 00:00 2024-03-26T04:37:48.152 INFO:teuthology.orchestra.run.smithi012.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-26T04:37:48.202 INFO:teuthology.orchestra.run.smithi012.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 246 kB/s | 172 kB 00:00 2024-03-26T04:37:48.228 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:37:48.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T04:37:48.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T04:37:48.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:37:48.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T04:37:48.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T04:37:48.261 INFO:teuthology.orchestra.run.smithi012.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-26T04:37:48.320 INFO:teuthology.orchestra.run.smithi012.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-26T04:37:48.362 INFO:teuthology.orchestra.run.smithi012.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-26T04:37:48.429 INFO:teuthology.orchestra.run.smithi012.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-26T04:37:48.470 INFO:teuthology.orchestra.run.smithi012.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2024-03-26T04:37:48.504 INFO:teuthology.orchestra.run.smithi012.stdout:(28/36): python3-cherrypy-18.6.1-2.el9.noarch.r 604 kB/s | 358 kB 00:00 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:48.582 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:37:48.613 INFO:teuthology.orchestra.run.smithi012.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-26T04:37:48.638 INFO:teuthology.orchestra.run.smithi012.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 377 kB/s | 79 kB 00:00 2024-03-26T04:37:48.747 INFO:teuthology.orchestra.run.smithi012.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-03-26T04:37:48.805 INFO:teuthology.orchestra.run.smithi012.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 468 kB/s | 90 kB 00:00 2024-03-26T04:37:48.904 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph 2024-03-26T04:37:48.914 INFO:teuthology.orchestra.run.smithi012.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 615 kB/s | 272 kB 00:00 2024-03-26T04:37:49.023 INFO:teuthology.orchestra.run.smithi012.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-26T04:37:49.106 INFO:teuthology.orchestra.run.smithi012.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 640 kB/s | 230 kB 00:00 2024-03-26T04:37:49.431 INFO:teuthology.orchestra.run.smithi012.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 683 kB/s | 427 kB 00:00 2024-03-26T04:37:49.437 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:37:49.437 INFO:teuthology.orchestra.run.smithi012.stdout:Total 6.9 MB/s | 30 MB 00:04 2024-03-26T04:37:49.461 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:37:49.491 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:44 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:37:49.506 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:37:49.506 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:37:49.652 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T04:37:49.655 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T04:37:49.656 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T04:37:49.657 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout:Install 36 Packages 2024-03-26T04:37:49.658 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:49.659 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 30 M 2024-03-26T04:37:49.659 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 107 M 2024-03-26T04:37:49.659 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:37:49.925 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:37:49.925 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:37:50.191 INFO:teuthology.orchestra.run.smithi186.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-26T04:37:50.324 INFO:teuthology.orchestra.run.smithi186.stdout:(2/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 6.1 MB/s | 2.3 MB 00:00 2024-03-26T04:37:51.158 INFO:teuthology.orchestra.run.smithi186.stdout:(3/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-26T04:37:51.191 INFO:teuthology.orchestra.run.smithi186.stdout:(4/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-26T04:37:51.217 INFO:teuthology.orchestra.run.smithi186.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 4.2 MB/s | 248 kB 00:00 2024-03-26T04:37:51.342 INFO:teuthology.orchestra.run.smithi186.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-26T04:37:51.367 INFO:teuthology.orchestra.run.smithi186.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-26T04:37:51.392 INFO:teuthology.orchestra.run.smithi186.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-26T04:37:51.395 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:37:51.417 INFO:teuthology.orchestra.run.smithi186.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-26T04:37:51.443 INFO:teuthology.orchestra.run.smithi186.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-26T04:37:51.450 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T04:37:51.468 INFO:teuthology.orchestra.run.smithi186.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T04:37:51.483 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T04:37:51.511 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T04:37:51.586 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T04:37:51.602 INFO:teuthology.orchestra.run.smithi186.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 7.2 MB/s | 1.1 MB 00:00 2024-03-26T04:37:51.619 INFO:teuthology.orchestra.run.smithi186.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 686 kB/s | 103 kB 00:00 2024-03-26T04:37:51.635 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T04:37:51.661 INFO:teuthology.orchestra.run.smithi186.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-26T04:37:51.668 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T04:37:51.686 INFO:teuthology.orchestra.run.smithi186.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-26T04:37:51.761 INFO:teuthology.orchestra.run.smithi186.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-03-26T04:37:51.764 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T04:37:51.795 INFO:teuthology.orchestra.run.smithi186.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-26T04:37:51.957 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T04:37:52.004 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T04:37:52.062 INFO:teuthology.orchestra.run.smithi186.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-03-26T04:37:52.078 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T04:37:52.131 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T04:37:52.145 INFO:teuthology.orchestra.run.smithi186.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 393 kB/s | 151 kB 00:00 2024-03-26T04:37:52.176 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T04:37:52.235 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T04:37:52.262 INFO:teuthology.orchestra.run.smithi186.stdout:(20/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-03-26T04:37:52.288 INFO:teuthology.orchestra.run.smithi186.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2024-03-26T04:37:52.307 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T04:37:52.355 INFO:teuthology.orchestra.run.smithi186.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-26T04:37:52.380 INFO:teuthology.orchestra.run.smithi186.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-26T04:37:52.406 INFO:teuthology.orchestra.run.smithi186.stdout:(24/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2024-03-26T04:37:52.430 INFO:teuthology.orchestra.run.smithi186.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-03-26T04:37:52.456 INFO:teuthology.orchestra.run.smithi186.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-26T04:37:52.498 INFO:teuthology.orchestra.run.smithi186.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-26T04:37:52.508 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T04:37:52.523 INFO:teuthology.orchestra.run.smithi186.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-26T04:37:52.557 INFO:teuthology.orchestra.run.smithi186.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 782 kB/s | 79 kB 00:00 2024-03-26T04:37:52.563 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T04:37:52.599 INFO:teuthology.orchestra.run.smithi186.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-26T04:37:52.604 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T04:37:52.646 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T04:37:52.675 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T04:37:52.732 INFO:teuthology.orchestra.run.smithi186.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 513 kB/s | 90 kB 00:00 2024-03-26T04:37:52.754 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T04:37:52.758 INFO:teuthology.orchestra.run.smithi186.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 225 kB/s | 36 kB 00:00 2024-03-26T04:37:52.785 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:37:52.816 INFO:teuthology.orchestra.run.smithi186.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 855 kB/s | 272 kB 00:00 2024-03-26T04:37:52.819 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T04:37:52.842 INFO:teuthology.orchestra.run.smithi186.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-26T04:37:52.852 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T04:37:52.895 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T04:37:52.908 INFO:teuthology.orchestra.run.smithi186.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-26T04:37:52.928 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T04:37:52.992 INFO:teuthology.orchestra.run.smithi186.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-03-26T04:37:52.996 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:37:52.996 INFO:teuthology.orchestra.run.smithi186.stdout:Total 9.0 MB/s | 30 MB 00:03 2024-03-26T04:37:53.020 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:37:53.021 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T04:37:53.060 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T04:37:53.064 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:37:53.064 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:37:53.135 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T04:37:53.275 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T04:37:53.400 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:37:53.494 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:37:53.494 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:37:53.567 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T04:37:53.606 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:37:53.632 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:37:53.632 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:53.632 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T04:37:53.632 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:37:53.632 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:37:53.632 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:54.833 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:37:54.903 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T04:37:54.936 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T04:37:54.964 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T04:37:55.035 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T04:37:55.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T04:37:55.129 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T04:37:55.148 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:37:55.175 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:37:55.175 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:55.175 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T04:37:55.175 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:37:55.175 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:37:55.175 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:55.233 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T04:37:55.308 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T04:37:55.357 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T04:37:55.429 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T04:37:55.483 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T04:37:55.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T04:37:55.570 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T04:37:55.629 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T04:37:55.640 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:37:55.664 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:37:55.664 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:55.664 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T04:37:55.664 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:37:55.664 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:37:55.664 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:55.822 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T04:37:55.867 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T04:37:55.899 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:37:55.911 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T04:37:55.922 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:37:55.923 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:55.923 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T04:37:55.923 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:37:55.923 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:37:55.923 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:55.949 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T04:37:55.979 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T04:37:55.990 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:37:56.057 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T04:37:56.089 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:37:56.123 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T04:37:56.155 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T04:37:56.190 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T04:37:56.224 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T04:37:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T04:37:56.355 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T04:37:56.429 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T04:37:56.566 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T04:37:56.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:37:56.856 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T04:37:56.885 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:37:56.911 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:37:56.911 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:56.911 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T04:37:56.911 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:37:56.911 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:37:56.911 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:57.758 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:37:57.758 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T04:37:57.759 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T04:37:57.760 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T04:37:57.761 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:37:57.761 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T04:37:57.761 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T04:37:57.761 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T04:37:57.761 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T04:37:57.761 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T04:37:58.353 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:37:58.379 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:37:58.380 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:58.380 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T04:37:58.380 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:37:58.380 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:37:58.380 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:58.794 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T04:37:58.794 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:58.794 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:37:58.794 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T04:37:58.795 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T04:37:58.796 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:37:58.797 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:37:58.820 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:37:58.843 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:37:58.843 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:58.843 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T04:37:58.843 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:37:58.844 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:37:58.844 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:59.071 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:37:59.094 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:37:59.094 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:37:59.094 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T04:37:59.095 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:37:59.095 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:37:59.095 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:37:59.153 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-base 2024-03-26T04:37:59.169 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:37:59.722 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:37:59.822 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:37:59.876 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:37:59.878 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:37:59.878 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:37:59.947 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install cephadm 2024-03-26T04:38:00.515 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:59 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:00.669 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:00.670 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:00.671 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:00.671 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:00.671 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 748 k 2024-03-26T04:38:00.671 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 754 k 2024-03-26T04:38:00.671 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T04:38:00.885 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T04:38:00.886 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T04:38:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:38:00.888 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T04:38:00.888 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T04:38:00.888 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T04:38:00.888 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T04:38:00.888 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T04:38:01.098 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-26T04:38:01.099 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:01.099 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-26T04:38:01.099 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:01.105 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:01.105 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:01.110 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:01.111 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:01.185 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:01.582 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:01.689 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T04:38:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:01.793 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T04:38:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T04:38:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:01.796 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:02.079 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:02.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base 2024-03-26T04:38:02.393 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:02.393 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:02.393 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:02.393 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:02.393 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:02.393 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:02.585 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T04:38:02.670 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:02.768 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:02.819 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:02.820 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:02.820 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:02.888 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install cephadm 2024-03-26T04:38:03.159 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:03.313 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:03.314 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 145 k 2024-03-26T04:38:03.315 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 436 k 2024-03-26T04:38:03.315 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:03.465 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:03.529 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 678 kB/s | 145 kB 00:00 2024-03-26T04:38:03.529 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:03.529 INFO:teuthology.orchestra.run.smithi012.stdout:Total 674 kB/s | 145 kB 00:00 2024-03-26T04:38:03.530 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:03.538 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:03.538 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:03.592 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:03.592 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:03.615 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:03.615 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 748 k 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 754 k 2024-03-26T04:38:03.616 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:03.734 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:03.805 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:38:03.828 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:38:03.828 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:38:03.829 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T04:38:03.829 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:04.050 INFO:teuthology.orchestra.run.smithi186.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-26T04:38:04.050 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:04.050 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-26T04:38:04.051 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:04.056 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:04.056 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:04.061 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:04.062 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:04.134 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:04.557 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:04.638 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:04.776 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:38:04.776 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:04.776 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:04.776 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:04.776 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:04.776 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:05.003 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:05.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr 2024-03-26T04:38:05.322 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:38:05.322 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:05.322 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:05.322 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:05.323 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:05.323 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:05.494 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T04:38:05.598 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:05.698 INFO:teuthology.orchestra.run.smithi012.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:05.753 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:05.754 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:05.754 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:05.813 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T04:38:06.078 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:06.230 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:06.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:06.232 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:06.232 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:06.232 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 145 k 2024-03-26T04:38:06.232 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 436 k 2024-03-26T04:38:06.232 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:06.380 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:05 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:06.536 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================== 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================== 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T04:38:06.537 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================== 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout:Install 12 Packages 2024-03-26T04:38:06.538 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:06.539 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 6.9 M 2024-03-26T04:38:06.539 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 26 M 2024-03-26T04:38:06.539 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:06.539 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 471 kB/s | 145 kB 00:00 2024-03-26T04:38:06.540 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:06.540 INFO:teuthology.orchestra.run.smithi186.stdout:Total 469 kB/s | 145 kB 00:00 2024-03-26T04:38:06.541 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:06.548 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:06.548 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:06.601 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:06.602 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:06.748 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:06.816 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:38:06.840 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:38:06.840 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:38:06.840 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T04:38:06.840 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:07.057 INFO:teuthology.orchestra.run.smithi012.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1775.g5e 153 kB/s | 31 kB 00:00 2024-03-26T04:38:07.116 INFO:teuthology.orchestra.run.smithi012.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 58 kB/s | 15 kB 00:00 2024-03-26T04:38:07.224 INFO:teuthology.orchestra.run.smithi012.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 4.8 MB/s | 1.8 MB 00:00 2024-03-26T04:38:07.249 INFO:teuthology.orchestra.run.smithi012.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-26T04:38:07.283 INFO:teuthology.orchestra.run.smithi012.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-26T04:38:07.450 INFO:teuthology.orchestra.run.smithi012.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-03-26T04:38:07.475 INFO:teuthology.orchestra.run.smithi012.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-03-26T04:38:07.525 INFO:teuthology.orchestra.run.smithi012.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-26T04:38:07.709 INFO:teuthology.orchestra.run.smithi012.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.9 MB/s | 2.0 MB 00:00 2024-03-26T04:38:07.734 INFO:teuthology.orchestra.run.smithi012.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 558 kB/s | 144 kB 00:00 2024-03-26T04:38:07.759 INFO:teuthology.orchestra.run.smithi012.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 132 kB/s | 31 kB 00:00 2024-03-26T04:38:07.776 INFO:teuthology.orchestra.run.smithi012.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-26T04:38:07.781 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:07.781 INFO:teuthology.orchestra.run.smithi012.stdout:Total 5.5 MB/s | 6.9 MB 00:01 2024-03-26T04:38:07.802 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:07.816 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:07.816 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:07.823 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:38:07.823 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:07.823 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:07.823 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:07.823 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:07.824 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:07.993 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:07.993 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:08.046 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr 2024-03-26T04:38:08.513 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T04:38:08.613 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:08.638 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T04:38:08.685 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T04:38:08.710 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:08.723 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T04:38:08.762 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:08.763 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:08.763 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:08.828 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T04:38:08.829 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T04:38:08.962 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T04:38:09.053 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T04:38:09.103 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T04:38:09.130 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:38:09.163 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T04:38:09.384 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:09.419 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T04:38:09.445 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:38:09.538 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T04:38:09.539 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-03-26T04:38:09.540 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:09.541 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.9 M 2024-03-26T04:38:09.541 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-03-26T04:38:09.541 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:09.894 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 64 kB/s | 15 kB 00:00 2024-03-26T04:38:09.919 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1775.g5e 119 kB/s | 31 kB 00:00 2024-03-26T04:38:10.045 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 4.6 MB/s | 1.8 MB 00:00 2024-03-26T04:38:10.145 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-26T04:38:10.170 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 949 kB/s | 238 kB 00:00 2024-03-26T04:38:10.209 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:38:10.209 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T04:38:10.210 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T04:38:10.354 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-03-26T04:38:10.521 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T04:38:10.569 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T04:38:10.570 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T04:38:10.570 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T04:38:10.570 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T04:38:10.570 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:10.570 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:10.654 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-26T04:38:10.713 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 526 kB/s | 31 kB 00:00 2024-03-26T04:38:10.780 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-03-26T04:38:10.794 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T04:38:10.838 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-26T04:38:11.214 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-26T04:38:11.219 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:11.220 INFO:teuthology.orchestra.run.smithi186.stdout:Total 4.1 MB/s | 6.9 MB 00:01 2024-03-26T04:38:11.238 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:11.255 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:11.255 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:11.361 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:10 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:11.438 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:11.438 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:11.517 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T04:38:11.519 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout:Install 12 Packages 2024-03-26T04:38:11.520 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:11.521 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 37 M 2024-03-26T04:38:11.521 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 209 M 2024-03-26T04:38:11.521 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:11.951 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:12.031 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T04:38:12.049 INFO:teuthology.orchestra.run.smithi012.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-03-26T04:38:12.066 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T04:38:12.083 INFO:teuthology.orchestra.run.smithi012.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-03-26T04:38:12.104 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T04:38:12.116 INFO:teuthology.orchestra.run.smithi012.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-26T04:38:12.143 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T04:38:12.175 INFO:teuthology.orchestra.run.smithi012.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-26T04:38:12.280 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T04:38:12.368 INFO:teuthology.orchestra.run.smithi012.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-26T04:38:12.411 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T04:38:12.434 INFO:teuthology.orchestra.run.smithi012.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-26T04:38:12.460 INFO:teuthology.orchestra.run.smithi012.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T04:38:12.506 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T04:38:12.556 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T04:38:12.568 INFO:teuthology.orchestra.run.smithi012.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-17 11 MB/s | 7.4 MB 00:00 2024-03-26T04:38:12.584 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:38:12.617 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T04:38:12.858 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T04:38:12.885 INFO:teuthology.orchestra.run.smithi012.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-26T04:38:12.910 INFO:teuthology.orchestra.run.smithi012.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 575 kB/s | 195 kB 00:00 2024-03-26T04:38:12.950 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:38:12.986 INFO:teuthology.orchestra.run.smithi012.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-26T04:38:13.136 INFO:teuthology.orchestra.run.smithi012.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 981 kB/s | 245 kB 00:00 2024-03-26T04:38:13.138 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:13.138 INFO:teuthology.orchestra.run.smithi012.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-26T04:38:13.178 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:13.202 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:13.203 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:13.574 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:13.575 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:13.731 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:38:13.731 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T04:38:13.732 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T04:38:14.093 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T04:38:14.093 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:14.093 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:14.093 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:14.094 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:14.315 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:14.350 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T04:38:14.376 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T04:38:14.407 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T04:38:14.513 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:38:14.552 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T04:38:14.738 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:38:14.768 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:38:14.847 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T04:38:14.910 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:09 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:15.060 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:15.061 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-26T04:38:15.061 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T04:38:15.062 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:15.063 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 37 M 2024-03-26T04:38:15.064 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 209 M 2024-03-26T04:38:15.064 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:15.530 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T04:38:15.554 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 228 kB/s | 32 kB 00:00 2024-03-26T04:38:15.582 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 722 kB/s | 17 kB 00:00 2024-03-26T04:38:15.613 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-26T04:38:15.660 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T04:38:15.839 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.1 MB/s | 3.0 MB 00:00 2024-03-26T04:38:15.922 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-26T04:38:16.106 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-26T04:38:16.131 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.0 MB/s | 217 kB 00:00 2024-03-26T04:38:16.581 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-26T04:38:16.756 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-26T04:38:16.798 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 195 kB 00:00 2024-03-26T04:38:16.978 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T04:38:17.265 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 482 kB/s | 245 kB 00:00 2024-03-26T04:38:17.399 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.7 MB/s | 7.4 MB 00:01 2024-03-26T04:38:17.401 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:17.401 INFO:teuthology.orchestra.run.smithi186.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-26T04:38:17.441 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:17.465 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:17.465 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:17.829 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:17.829 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:18.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T04:38:18.083 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:38:18.540 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:18.622 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T04:38:18.661 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T04:38:18.755 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:38:18.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T04:38:18.960 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:38:18.989 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:38:19.065 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T04:38:19.733 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T04:38:19.851 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:38:20.514 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:38:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T04:38:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T04:38:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T04:38:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T04:38:20.515 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T04:38:20.918 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:20.919 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:21.092 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T04:38:21.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-rook 2024-03-26T04:38:21.848 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:20 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:22.005 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T04:38:22.006 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2024-03-26T04:38:22.007 INFO:teuthology.orchestra.run.smithi012.stdout:Install 13 Packages 2024-03-26T04:38:22.008 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:22.008 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 2.0 M 2024-03-26T04:38:22.008 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 26 M 2024-03-26T04:38:22.008 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:22.121 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T04:38:22.147 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:38:22.491 INFO:teuthology.orchestra.run.smithi012.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 191 kB/s | 24 kB 00:00 2024-03-26T04:38:22.516 INFO:teuthology.orchestra.run.smithi012.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-26T04:38:22.558 INFO:teuthology.orchestra.run.smithi012.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-26T04:38:22.583 INFO:teuthology.orchestra.run.smithi012.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-26T04:38:22.608 INFO:teuthology.orchestra.run.smithi012.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T04:38:22.634 INFO:teuthology.orchestra.run.smithi012.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-26T04:38:22.659 INFO:teuthology.orchestra.run.smithi012.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 167 kB/s | 49 kB 00:00 2024-03-26T04:38:22.759 INFO:teuthology.orchestra.run.smithi012.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-26T04:38:22.819 INFO:teuthology.orchestra.run.smithi012.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-03-26T04:38:22.868 INFO:teuthology.orchestra.run.smithi012.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 970 kB/s | 203 kB 00:00 2024-03-26T04:38:22.893 INFO:teuthology.orchestra.run.smithi012.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-26T04:38:22.927 INFO:teuthology.orchestra.run.smithi012.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-26T04:38:23.019 INFO:teuthology.orchestra.run.smithi012.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-26T04:38:23.020 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:23.021 INFO:teuthology.orchestra.run.smithi012.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-26T04:38:23.021 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:23.035 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:23.035 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:23.130 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:23.130 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:23.657 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:23.832 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T04:38:23.888 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T04:38:23.943 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T04:38:23.980 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T04:38:24.021 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T04:38:24.118 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T04:38:24.236 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T04:38:24.289 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T04:38:24.684 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:38:24.684 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T04:38:24.684 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T04:38:24.684 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:38:24.684 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T04:38:24.685 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T04:38:25.063 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T04:38:25.119 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T04:38:25.153 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T04:38:25.172 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:25.173 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:25.201 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T04:38:25.231 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:38:25.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-rook 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T04:38:26.019 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T04:38:26.020 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T04:38:26.020 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T04:38:26.020 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T04:38:26.020 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T04:38:26.020 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T04:38:26.020 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T04:38:26.073 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:21 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:26.224 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:26.225 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-26T04:38:26.225 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:26.225 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-26T04:38:26.225 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:26.225 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T04:38:26.226 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T04:38:26.229 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout:Install 13 Packages 2024-03-26T04:38:26.230 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:26.231 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.0 M 2024-03-26T04:38:26.231 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-03-26T04:38:26.231 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:26.390 INFO:teuthology.orchestra.run.smithi186.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 205 kB/s | 24 kB 00:00 2024-03-26T04:38:26.409 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T04:38:26.409 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T04:38:26.410 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T04:38:26.411 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T04:38:26.411 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:26.411 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:26.416 INFO:teuthology.orchestra.run.smithi186.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-26T04:38:26.466 INFO:teuthology.orchestra.run.smithi186.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-26T04:38:26.500 INFO:teuthology.orchestra.run.smithi186.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T04:38:26.525 INFO:teuthology.orchestra.run.smithi186.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-26T04:38:26.542 INFO:teuthology.orchestra.run.smithi186.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-26T04:38:26.559 INFO:teuthology.orchestra.run.smithi186.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 171 kB/s | 49 kB 00:00 2024-03-26T04:38:26.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T04:38:26.934 INFO:teuthology.orchestra.run.smithi186.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2024-03-26T04:38:26.951 INFO:teuthology.orchestra.run.smithi186.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 76 kB/s | 32 kB 00:00 2024-03-26T04:38:27.143 INFO:teuthology.orchestra.run.smithi186.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 309 kB/s | 59 kB 00:00 2024-03-26T04:38:27.209 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:27.210 INFO:teuthology.orchestra.run.smithi186.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 311 kB/s | 203 kB 00:00 2024-03-26T04:38:27.293 INFO:teuthology.orchestra.run.smithi186.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-26T04:38:27.365 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:27.366 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2024-03-26T04:38:27.366 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:27.366 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2024-03-26T04:38:27.366 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================== 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout:Install 6 Packages 2024-03-26T04:38:27.367 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:27.368 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 6.8 M 2024-03-26T04:38:27.368 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 33 M 2024-03-26T04:38:27.368 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:27.518 INFO:teuthology.orchestra.run.smithi186.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-26T04:38:27.523 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:27.523 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-26T04:38:27.524 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:27.537 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:27.537 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:27.630 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:27.632 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:27.836 INFO:teuthology.orchestra.run.smithi012.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-26T04:38:27.861 INFO:teuthology.orchestra.run.smithi012.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 882 kB/s | 248 kB 00:00 2024-03-26T04:38:27.928 INFO:teuthology.orchestra.run.smithi012.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 396 kB/s | 138 kB 00:00 2024-03-26T04:38:28.129 INFO:teuthology.orchestra.run.smithi012.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 216 kB/s | 58 kB 00:00 2024-03-26T04:38:28.144 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:28.187 INFO:teuthology.orchestra.run.smithi012.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 197 kB/s | 51 kB 00:00 2024-03-26T04:38:28.317 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T04:38:28.379 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T04:38:28.442 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T04:38:28.471 INFO:teuthology.orchestra.run.smithi012.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 865 kB/s | 548 kB 00:00 2024-03-26T04:38:28.472 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:28.472 INFO:teuthology.orchestra.run.smithi012.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-26T04:38:28.480 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T04:38:28.487 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:28.499 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:28.499 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:28.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T04:38:28.597 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:28.598 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:28.615 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T04:38:28.725 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T04:38:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T04:38:28.901 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T04:38:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T04:38:29.516 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T04:38:29.569 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T04:38:29.603 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T04:38:29.651 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T04:38:29.673 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:38:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T04:38:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T04:38:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T04:38:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T04:38:30.425 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T04:38:30.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T04:38:30.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T04:38:30.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T04:38:30.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T04:38:30.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T04:38:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:38:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T04:38:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T04:38:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T04:38:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T04:38:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T04:38:30.818 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T04:38:30.818 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T04:38:30.819 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T04:38:30.820 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T04:38:30.820 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T04:38:30.820 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T04:38:30.820 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:30.820 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:31.005 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:31.054 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T04:38:31.251 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-fuse 2024-03-26T04:38:31.617 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:31.768 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:31.769 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:31.770 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-03-26T04:38:31.770 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-03-26T04:38:31.770 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:31.770 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.8 M 2024-03-26T04:38:31.770 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 33 M 2024-03-26T04:38:31.770 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:31.821 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:31.977 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout:Install 2 Packages 2024-03-26T04:38:31.978 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:31.979 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 967 k 2024-03-26T04:38:31.979 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 2.8 M 2024-03-26T04:38:31.979 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:32.213 INFO:teuthology.orchestra.run.smithi012.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 336 kB/s | 79 kB 00:00 2024-03-26T04:38:32.262 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 579 kB/s | 138 kB 00:00 2024-03-26T04:38:32.363 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 731 kB/s | 248 kB 00:00 2024-03-26T04:38:32.505 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-26T04:38:32.597 INFO:teuthology.orchestra.run.smithi012.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 1.4 MB/s | 888 kB 00:00 2024-03-26T04:38:32.598 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:32.599 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.5 MB/s | 967 kB 00:00 2024-03-26T04:38:32.599 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:32.621 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:32.621 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:32.639 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 211 kB/s | 58 kB 00:00 2024-03-26T04:38:32.664 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-26T04:38:32.689 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 279 kB/s | 51 kB 00:00 2024-03-26T04:38:32.692 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:32.692 INFO:teuthology.orchestra.run.smithi186.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2024-03-26T04:38:32.698 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:32.698 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:32.705 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:32.716 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:32.716 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:32.807 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:32.807 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:32.878 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:33.041 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T04:38:33.100 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:33.115 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:38:33.271 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T04:38:33.324 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T04:38:33.768 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:38:33.768 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T04:38:33.857 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T04:38:33.935 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T04:38:34.005 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T04:38:34.036 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:38:34.078 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T04:38:34.079 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:34.079 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:34.079 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T04:38:34.079 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:34.079 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:34.345 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-volume 2024-03-26T04:38:34.835 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:38:34.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T04:38:34.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T04:38:34.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T04:38:34.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T04:38:34.835 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T04:38:34.918 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:35.074 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout:Install 2 Packages 2024-03-26T04:38:35.075 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:35.076 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 332 k 2024-03-26T04:38:35.076 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 1.5 M 2024-03-26T04:38:35.076 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:35.159 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T04:38:35.159 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:35.159 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:35.160 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:35.195 INFO:teuthology.orchestra.run.smithi012.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 579 kB/s | 69 kB 00:00 2024-03-26T04:38:35.320 INFO:teuthology.orchestra.run.smithi012.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-26T04:38:35.321 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:35.321 INFO:teuthology.orchestra.run.smithi012.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-26T04:38:35.322 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:35.330 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:35.331 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:35.367 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:35.368 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:35.394 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse 2024-03-26T04:38:35.520 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:35.707 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T04:38:35.747 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:38:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:38:35.762 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:38:35.763 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T04:38:35.763 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:35.962 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:38:36.112 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 967 k 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.8 M 2024-03-26T04:38:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:36.233 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 663 kB/s | 79 kB 00:00 2024-03-26T04:38:36.449 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 2.6 MB/s | 888 kB 00:00 2024-03-26T04:38:36.451 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:36.451 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.8 MB/s | 967 kB 00:00 2024-03-26T04:38:36.451 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:36.471 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:36.471 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:36.524 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T04:38:36.547 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:36.547 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:36.730 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:36.878 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T04:38:36.879 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:36.879 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:36.879 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:36.879 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T04:38:36.879 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:36.879 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:36.884 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T04:38:36.950 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:38:37.193 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados-devel 2024-03-26T04:38:37.638 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:38:37.638 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T04:38:37.768 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:37.924 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:37.925 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:37.926 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 127 k 2024-03-26T04:38:37.926 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 456 k 2024-03-26T04:38:37.926 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:37.952 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T04:38:37.953 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:37.953 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:37.953 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T04:38:37.953 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:37.953 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:38.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-volume 2024-03-26T04:38:38.241 INFO:teuthology.orchestra.run.smithi012.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 402 kB/s | 127 kB 00:00 2024-03-26T04:38:38.242 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:38.242 INFO:teuthology.orchestra.run.smithi012.stdout:Total 400 kB/s | 127 kB 00:00 2024-03-26T04:38:38.242 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:38.248 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:38.248 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:38.306 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:38.307 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:38.461 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:38.568 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:38.802 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:38.953 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-26T04:38:38.954 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 332 k 2024-03-26T04:38:38.955 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.5 M 2024-03-26T04:38:38.956 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:39.008 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:39.073 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 587 kB/s | 69 kB 00:00 2024-03-26T04:38:39.223 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 987 kB/s | 264 kB 00:00 2024-03-26T04:38:39.224 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:39.224 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-26T04:38:39.225 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:39.233 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:39.233 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:39.272 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:39.272 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:39.318 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:39.318 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:39.319 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:39.319 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:39.319 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:39.319 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:39.422 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:39.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs2 2024-03-26T04:38:39.598 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T04:38:39.634 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:38:39.649 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:38:39.649 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:38:39.649 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T04:38:39.650 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:40.121 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:40.222 INFO:teuthology.orchestra.run.smithi012.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:40.277 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:40.279 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:40.279 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:40.339 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install libcephfs-devel 2024-03-26T04:38:40.368 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T04:38:40.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T04:38:40.727 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:40.728 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:40.728 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:38:40.728 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T04:38:40.728 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:40.728 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:40.913 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:41.029 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel 2024-03-26T04:38:41.069 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:41.071 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 31 k 2024-03-26T04:38:41.071 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 140 k 2024-03-26T04:38:41.071 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:41.328 INFO:teuthology.orchestra.run.smithi012.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 122 kB/s | 31 kB 00:00 2024-03-26T04:38:41.328 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:41.328 INFO:teuthology.orchestra.run.smithi012.stdout:Total 121 kB/s | 31 kB 00:00 2024-03-26T04:38:41.329 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:41.332 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:41.332 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:41.369 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:41.370 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:41.482 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:41.596 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:41.603 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:41.754 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:41.755 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:41.756 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 127 k 2024-03-26T04:38:41.756 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 456 k 2024-03-26T04:38:41.756 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:41.959 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:42.063 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 413 kB/s | 127 kB 00:00 2024-03-26T04:38:42.063 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:42.064 INFO:teuthology.orchestra.run.smithi186.stdout:Total 411 kB/s | 127 kB 00:00 2024-03-26T04:38:42.064 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:42.069 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:42.069 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:42.127 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:42.127 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:42.236 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:42.236 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:42.236 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:42.278 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:42.389 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:42.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librados2 2024-03-26T04:38:42.821 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:43.002 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:43.097 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:43.097 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:43.097 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:43.097 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:43.097 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:43.098 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:43.103 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:43.159 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:43.160 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:43.160 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:43.204 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install librbd1 2024-03-26T04:38:43.306 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2 2024-03-26T04:38:43.783 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:43.883 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:43.885 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:43.937 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:43.939 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:43.939 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:43.988 INFO:teuthology.orchestra.run.smithi186.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:43.992 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rados 2024-03-26T04:38:44.043 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:44.045 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:44.045 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:44.107 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs-devel 2024-03-26T04:38:44.565 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:44.665 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:44.669 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:44.719 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:44.721 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:44.721 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:44.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rgw 2024-03-26T04:38:44.820 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:44.821 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:44.822 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:44.822 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 31 k 2024-03-26T04:38:44.822 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 140 k 2024-03-26T04:38:44.822 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:44.992 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 184 kB/s | 31 kB 00:00 2024-03-26T04:38:44.992 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:44.992 INFO:teuthology.orchestra.run.smithi186.stdout:Total 183 kB/s | 31 kB 00:00 2024-03-26T04:38:44.993 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:44.996 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:44.996 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:45.025 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:45.025 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:45.146 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:45.241 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:45.339 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:45.439 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:45.493 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:45.495 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:45.495 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:45.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-cephfs 2024-03-26T04:38:45.593 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:45.873 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:45.873 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:45.873 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:45.873 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:45.873 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:45.873 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:46.017 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2 2024-03-26T04:38:46.112 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:46.213 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:46.268 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:46.269 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:46.269 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:46.322 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install python3-rbd 2024-03-26T04:38:46.583 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:46.680 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:46.731 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:46.733 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:46.733 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:46.780 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1 2024-03-26T04:38:46.896 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:46.995 INFO:teuthology.orchestra.run.smithi012.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:47.049 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-03-26T04:38:47.051 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:47.096 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-fuse 2024-03-26T04:38:47.342 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:47.439 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:47.491 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:47.492 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:47.492 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:47.550 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados 2024-03-26T04:38:47.666 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:47.820 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:47.820 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:47.821 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:47.822 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 85 k 2024-03-26T04:38:47.822 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 231 k 2024-03-26T04:38:47.822 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:48.121 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:48.206 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 222 kB/s | 85 kB 00:00 2024-03-26T04:38:48.206 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:48.207 INFO:teuthology.orchestra.run.smithi012.stdout:Total 221 kB/s | 85 kB 00:00 2024-03-26T04:38:48.207 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:48.214 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:48.214 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:48.217 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:48.268 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:48.269 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:48.270 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:48.272 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:48.273 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:48.322 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw 2024-03-26T04:38:48.443 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:48.557 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:48.889 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:48.959 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:48.986 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:49.037 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:49.039 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:49.039 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:49.090 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs 2024-03-26T04:38:49.268 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:49.268 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:49.268 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:49.268 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:49.268 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:49.268 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:49.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-mirror 2024-03-26T04:38:49.651 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:49.754 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:49.808 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:49.809 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:49.809 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:49.865 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd 2024-03-26T04:38:50.070 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:50.223 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:50.224 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:50.224 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repo Size 2024-03-26T04:38:50.224 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:50.224 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:50.224 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 3.1 M 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 13 M 2024-03-26T04:38:50.225 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:50.430 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:50.527 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:38:50.578 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:50.580 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-03-26T04:38:50.580 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:50.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse 2024-03-26T04:38:50.975 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 4.1 MB/s | 3.1 MB 00:00 2024-03-26T04:38:50.976 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:50.976 INFO:teuthology.orchestra.run.smithi012.stdout:Total 4.1 MB/s | 3.1 MB 00:00 2024-03-26T04:38:50.977 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:50.987 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:50.987 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:51.064 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:51.065 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:51.197 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:51.345 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:51.346 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:51.346 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-26T04:38:51.346 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:51.346 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 85 k 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 231 k 2024-03-26T04:38:51.347 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:51.514 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:51.545 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 431 kB/s | 85 kB 00:00 2024-03-26T04:38:51.545 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:51.546 INFO:teuthology.orchestra.run.smithi186.stdout:Total 428 kB/s | 85 kB 00:00 2024-03-26T04:38:51.546 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:51.553 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:51.553 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:51.584 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:51.608 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:51.608 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:38:51.609 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T04:38:51.609 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:38:51.609 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:38:51.609 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:51.609 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:51.610 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:51.744 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:51.852 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:52.221 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:52.485 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:52.485 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:52.485 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:52.485 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:52.486 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:52.486 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:52.709 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror 2024-03-26T04:38:52.766 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:52.766 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:52.766 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:52.766 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:52.767 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:52.767 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:52.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install rbd-nbd 2024-03-26T04:38:53.297 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:53.452 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:53.453 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:53.453 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-03-26T04:38:53.453 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 3.1 M 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 13 M 2024-03-26T04:38:53.454 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:53.530 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 04:37:01 AM UTC. 2024-03-26T04:38:53.685 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-03-26T04:38:53.685 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:Install 1 Package 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 171 k 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 494 k 2024-03-26T04:38:53.686 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-03-26T04:38:53.937 INFO:teuthology.orchestra.run.smithi012.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 681 kB/s | 171 kB 00:00 2024-03-26T04:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout:Total 676 kB/s | 171 kB 00:00 2024-03-26T04:38:53.938 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-03-26T04:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-03-26T04:38:53.947 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-03-26T04:38:54.006 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-03-26T04:38:54.006 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-03-26T04:38:54.156 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-03-26T04:38:54.263 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:54.316 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 3.6 MB/s | 3.1 MB 00:00 2024-03-26T04:38:54.316 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:54.317 INFO:teuthology.orchestra.run.smithi186.stdout:Total 3.6 MB/s | 3.1 MB 00:00 2024-03-26T04:38:54.317 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:54.327 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:54.327 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:54.682 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:54.821 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:54.892 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:54.917 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:54.917 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:38:54.917 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T04:38:54.917 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:38:54.917 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:38:54.917 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:54.975 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:54.975 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:54.975 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-03-26T04:38:54.975 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:54.975 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:38:54.975 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-03-26T04:38:55.217 DEBUG:teuthology.parallel:result is None 2024-03-26T04:38:56.112 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:56.112 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:56.112 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:56.112 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:56.112 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:56.112 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:56.352 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd 2024-03-26T04:38:56.930 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 04:37:05 AM UTC. 2024-03-26T04:38:57.080 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-26T04:38:57.081 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:57.081 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-26T04:38:57.081 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:57.081 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-26T04:38:57.081 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 171 k 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 494 k 2024-03-26T04:38:57.082 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-26T04:38:57.495 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 414 kB/s | 171 kB 00:00 2024-03-26T04:38:57.496 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:38:57.496 INFO:teuthology.orchestra.run.smithi186.stdout:Total 412 kB/s | 171 kB 00:00 2024-03-26T04:38:57.496 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-26T04:38:57.505 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-26T04:38:57.505 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-26T04:38:57.561 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-26T04:38:57.562 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-26T04:38:57.716 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-26T04:38:57.827 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:58.257 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:58.529 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:38:58.530 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:58.530 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-26T04:38:58.530 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:38:58.530 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:38:58.530 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-26T04:38:58.754 DEBUG:teuthology.parallel:result is None 2024-03-26T04:38:58.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:38:58.974 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T04:38:58.998 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T04:38:58.999 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T04:38:58.999 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T04:38:59.001 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:38:59.148 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T04:38:59.168 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T04:38:59.168 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T04:38:59.169 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T04:38:59.170 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T04:38:59.171 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:38:59.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T04:38:59.203 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:38:59.203 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T04:38:59.236 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T04:38:59.236 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:38:59.236 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T04:38:59.273 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T04:38:59.344 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:38:59.344 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T04:38:59.370 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T04:38:59.447 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T04:38:59.447 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:38:59.447 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T04:38:59.478 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T04:38:59.547 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:38:59.547 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T04:38:59.574 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T04:38:59.641 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T04:38:59.641 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:38:59.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T04:38:59.671 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T04:38:59.740 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:38:59.740 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T04:38:59.767 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T04:38:59.834 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T04:38:59.930 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'cephadm_mode': 'root'} 2024-03-26T04:38:59.930 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:38:59.931 INFO:tasks.cephadm:Cluster fsid is c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:38:59.931 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T04:38:59.931 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.12', 'mon.c': '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]', 'mon.b': '172.21.15.186'} 2024-03-26T04:38:59.931 INFO:tasks.cephadm:First mon is mon.a on smithi012 2024-03-26T04:38:59.931 INFO:tasks.cephadm:First mgr is y 2024-03-26T04:38:59.931 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T04:38:59.931 DEBUG:teuthology.orchestra.run.smithi012:> sudo hostname $(hostname -s) 2024-03-26T04:38:59.962 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-03-26T04:38:59.989 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T04:38:59.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:00.131 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54690/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1775-g5e70124b', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1775.g5e70124b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-25 18:23:16.641218', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T04:39:00.276 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 5e70124b46c2a671efcaae4690cd9ea72670710a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:00.277 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm 2024-03-26T04:39:00.278 DEBUG:teuthology.orchestra.run.smithi012:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:39:00.735 INFO:teuthology.orchestra.run.smithi012.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:39 /home/ubuntu/cephtest/cephadm 2024-03-26T04:39:00.735 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:39:01.355 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 766713 Mar 26 04:39 /home/ubuntu/cephtest/cephadm 2024-03-26T04:39:01.355 DEBUG:teuthology.orchestra.run.smithi012:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:39:01.374 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:39:01.404 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T04:39:01.404 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:39:01.416 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:39:02.279 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:39:02.356 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:39:34.311 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-03-26T04:39:34.311 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:39:34.311 INFO:teuthology.orchestra.run.smithi012.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:39:34.312 INFO:teuthology.orchestra.run.smithi012.stdout: "repo_digests": [ 2024-03-26T04:39:34.312 INFO:teuthology.orchestra.run.smithi012.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:39:34.312 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-03-26T04:39:34.312 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-03-26T04:39:35.470 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-26T04:39:35.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph 2024-03-26T04:39:35.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-03-26T04:39:35.558 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /etc/ceph 2024-03-26T04:39:35.599 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-03-26T04:39:35.626 INFO:tasks.cephadm:Writing seed config... 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T04:39:35.627 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T04:39:35.628 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-26T04:39:35.628 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-26T04:39:35.628 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T04:39:35.628 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:39:35.628 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T04:39:35.656 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c3b00422-eb2a-11ee-95d0-87774f69a715 mon election default strategy = 3 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-26T04:39:35.656 DEBUG:teuthology.orchestra.run.smithi012:mon.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T04:39:35.699 DEBUG:teuthology.orchestra.run.smithi012:mgr.y> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T04:39:35.741 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T04:39:35.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.12 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:39:35.943 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:39:35.944 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', 'c3b00422-eb2a-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.12', '--skip-admin-label'] 2024-03-26T04:39:35.968 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:39:35.969 INFO:teuthology.orchestra.run.smithi012.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T04:39:35.969 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying podman|docker is present... 2024-03-26T04:39:35.992 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:39:35.992 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying lvm2 is present... 2024-03-26T04:39:35.992 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying time synchronization is in place... 2024-03-26T04:39:36.000 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:39:36.000 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:39:36.006 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:39:36.006 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-26T04:39:36.012 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-03-26T04:39:36.018 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-03-26T04:39:36.018 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:39:36.018 INFO:teuthology.orchestra.run.smithi012.stdout:Repeating the final host check... 2024-03-26T04:39:36.040 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:39:36.040 INFO:teuthology.orchestra.run.smithi012.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T04:39:36.041 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl is present 2024-03-26T04:39:36.041 INFO:teuthology.orchestra.run.smithi012.stdout:lvcreate is present 2024-03-26T04:39:36.047 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:39:36.047 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:39:36.052 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:39:36.053 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-26T04:39:36.059 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-03-26T04:39:36.065 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-03-26T04:39:36.065 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:39:36.065 INFO:teuthology.orchestra.run.smithi012.stdout:Host looks OK 2024-03-26T04:39:36.065 INFO:teuthology.orchestra.run.smithi012.stdout:Cluster fsid: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:36.065 INFO:teuthology.orchestra.run.smithi012.stdout:Acquiring lock 139866467620464 on /run/cephadm/c3b00422-eb2a-11ee-95d0-87774f69a715.lock 2024-03-26T04:39:36.065 INFO:teuthology.orchestra.run.smithi012.stdout:Lock 139866467620464 acquired on /run/cephadm/c3b00422-eb2a-11ee-95d0-87774f69a715.lock 2024-03-26T04:39:36.066 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 3300 ... 2024-03-26T04:39:36.066 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 6789 ... 2024-03-26T04:39:36.066 INFO:teuthology.orchestra.run.smithi012.stdout:Base mon IP(s) is [172.21.15.12:3300, 172.21.15.12:6789], mon addrv is [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-03-26T04:39:36.069 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.12 metric 100 2024-03-26T04:39:36.069 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.12 metric 100 2024-03-26T04:39:36.071 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T04:39:36.071 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T04:39:36.071 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T04:39:36.074 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T04:39:36.074 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T04:39:36.074 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:39:36.074 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T04:39:36.074 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-03-26T04:39:36.074 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:39:36.075 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-03-26T04:39:36.075 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-03-26T04:39:36.075 INFO:teuthology.orchestra.run.smithi012.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T04:39:36.075 INFO:teuthology.orchestra.run.smithi012.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T04:39:36.076 INFO:teuthology.orchestra.run.smithi012.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:39:36.547 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:39:36.548 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:39:36.548 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T04:39:36.548 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T04:39:36.548 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T04:39:36.548 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:39:36.548 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T04:39:37.420 INFO:teuthology.orchestra.run.smithi012.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:39:37.420 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:39:37.420 INFO:teuthology.orchestra.run.smithi012.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T04:39:38.275 INFO:teuthology.orchestra.run.smithi012.stdout:stat: stdout 167 167 2024-03-26T04:39:38.275 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial keys... 2024-03-26T04:39:39.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCKUQJmgOdmJxAA9Uf0TS6Pn/zmw9GMf7pOHQ== 2024-03-26T04:39:39.972 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCLUQJmoL1ZHhAAHwCZLFBBVHwd+jEsanHFQg== 2024-03-26T04:39:40.877 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQCMUQJmPknSEBAA9MlnKCOB19c/kLk4I/RuIA== 2024-03-26T04:39:40.879 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial monmap... 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool for a [v2:172.21.15.12:3300,v1:172.21.15.12:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: set fsid to c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:39:41.918 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:39:41.919 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mon... 2024-03-26T04:39:42.814 INFO:teuthology.orchestra.run.smithi012.stdout:create mon.a on 2024-03-26T04:39:43.030 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T04:39:43.181 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T04:39:43.378 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715.target. 2024-03-26T04:39:43.379 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715.target. 2024-03-26T04:39:43.626 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:39:43.627 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T04:39:43.789 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715.target.wants/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@.service. 2024-03-26T04:39:44.357 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:39:44.357 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-26T04:39:44.364 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:39:44.364 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-26T04:39:44.364 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-26T04:39:44.364 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:39:44.364 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon to start... 2024-03-26T04:39:44.364 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon... 2024-03-26T04:39:45.433 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:45 smithi012 ceph-mon[29493]: from='client.? 172.21.15.12:0/1256039424' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout id: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout services: 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.567904s) 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout data: 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T04:39:45.441 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.811+0000 7f58a027d700 1 Processor -- start 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.811+0000 7f58a027d700 1 -- start start 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.811+0000 7f58a027d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f5898108ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.811+0000 7f58a027d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58980796b0 con 0x7f58981068b0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589e019700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f5898108ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589e019700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f5898108ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52242/0 (socket says 172.21.15.12:52242) 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589e019700 1 -- 172.21.15.12:0/3211630392 learned_addr learned my addr 172.21.15.12:0/3211630392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589e019700 1 -- 172.21.15.12:0/3211630392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58981091e0 con 0x7f58981068b0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589e019700 1 --2- 172.21.15.12:0/3211630392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f5898108ca0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5888009ab0 tx=0x7f5888009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e89eafe67447e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589d017700 1 -- 172.21.15.12:0/3211630392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5888004160 con 0x7f58981068b0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589d017700 1 -- 172.21.15.12:0/3211630392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58880042c0 con 0x7f58981068b0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.812+0000 7f589d017700 1 -- 172.21.15.12:0/3211630392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5888004580 con 0x7f58981068b0 2024-03-26T04:39:45.442 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 -- 172.21.15.12:0/3211630392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 msgr2=0x7f5898108ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 --2- 172.21.15.12:0/3211630392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f5898108ca0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5888009ab0 tx=0x7f5888009dc0 comp rx=0 tx=0).stop 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 -- 172.21.15.12:0/3211630392 shutdown_connections 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 --2- 172.21.15.12:0/3211630392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f5898108ca0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 -- 172.21.15.12:0/3211630392 >> 172.21.15.12:0/3211630392 conn(0x7f58981002c0 msgr2=0x7f58981026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 -- 172.21.15.12:0/3211630392 shutdown_connections 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.813+0000 7f58a027d700 1 -- 172.21.15.12:0/3211630392 wait complete. 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f58a027d700 1 Processor -- start 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f58a027d700 1 -- start start 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f58a027d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f58981992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f58a027d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58981997e0 con 0x7f58981068b0 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f589e019700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f58981992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f589e019700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f58981992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52258/0 (socket says 172.21.15.12:52258) 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f589e019700 1 -- 172.21.15.12:0/1256039424 learned_addr learned my addr 172.21.15.12:0/1256039424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:45.443 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f589e019700 1 -- 172.21.15.12:0/1256039424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5888009550 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.814+0000 7f589e019700 1 --2- 172.21.15.12:0/1256039424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f58981992a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5888009130 tx=0x7f5888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58880036a0 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888003800 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5888003ae0 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58981999e0 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5898199e80 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5888022070 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f588801b790 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.815+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58981930e0 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.818+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5888025d30 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.891+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f58981096a0 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.891+0000 7f588f7fe700 1 -- 172.21.15.12:0/1256039424 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f5888003c40 con 0x7f58981068b0 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 msgr2=0x7f58981992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 --2- 172.21.15.12:0/1256039424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f58981992a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5888009130 tx=0x7f5888004750 comp rx=0 tx=0).stop 2024-03-26T04:39:45.444 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 shutdown_connections 2024-03-26T04:39:45.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 --2- 172.21.15.12:0/1256039424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58981068b0 0x7f58981992a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:45.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 >> 172.21.15.12:0/1256039424 conn(0x7f58981002c0 msgr2=0x7f5898102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:45.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 shutdown_connections 2024-03-26T04:39:45.445 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:44.892+0000 7f58a027d700 1 -- 172.21.15.12:0/1256039424 wait complete. 2024-03-26T04:39:45.445 INFO:teuthology.orchestra.run.smithi012.stdout:mon is available 2024-03-26T04:39:45.445 INFO:teuthology.orchestra.run.smithi012.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T04:39:46.505 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:46 smithi012 ceph-mon[29493]: from='client.? 172.21.15.12:0/234107826' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:39:46.506 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:46 smithi012 ceph-mon[29493]: from='client.? 172.21.15.12:0/234107826' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:39:46.509 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:39:46.515 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe5e383700 1 Processor -- start 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe5e383700 1 -- start start 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe5e383700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58108cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe5e383700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580796b0 con 0x7efe581068d0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe57fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58108cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe57fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58108cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52268/0 (socket says 172.21.15.12:52268) 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe57fff700 1 -- 172.21.15.12:0/628925519 learned_addr learned my addr 172.21.15.12:0/628925519 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.843+0000 7efe57fff700 1 -- 172.21.15.12:0/628925519 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe58109200 con 0x7efe581068d0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe57fff700 1 --2- 172.21.15.12:0/628925519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58108cc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efe480098a0 tx=0x7efe48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4964a015b53937c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe56ffd700 1 -- 172.21.15.12:0/628925519 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe48004160 con 0x7efe581068d0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe56ffd700 1 -- 172.21.15.12:0/628925519 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe480042c0 con 0x7efe581068d0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe56ffd700 1 -- 172.21.15.12:0/628925519 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe480045a0 con 0x7efe581068d0 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 -- 172.21.15.12:0/628925519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 msgr2=0x7efe58108cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 --2- 172.21.15.12:0/628925519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58108cc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efe480098a0 tx=0x7efe48009bb0 comp rx=0 tx=0).stop 2024-03-26T04:39:46.516 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 -- 172.21.15.12:0/628925519 shutdown_connections 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 --2- 172.21.15.12:0/628925519 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58108cc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 -- 172.21.15.12:0/628925519 >> 172.21.15.12:0/628925519 conn(0x7efe581002a0 msgr2=0x7efe581026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 -- 172.21.15.12:0/628925519 shutdown_connections 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.844+0000 7efe5e383700 1 -- 172.21.15.12:0/628925519 wait complete. 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe5e383700 1 Processor -- start 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe5e383700 1 -- start start 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe5e383700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58106150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe5e383700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58106690 con 0x7efe581068d0 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe57fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58106150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe57fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58106150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52278/0 (socket says 172.21.15.12:52278) 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.845+0000 7efe57fff700 1 -- 172.21.15.12:0/234107826 learned_addr learned my addr 172.21.15.12:0/234107826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe57fff700 1 -- 172.21.15.12:0/234107826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe48009550 con 0x7efe581068d0 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe57fff700 1 --2- 172.21.15.12:0/234107826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58106150 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efe58078af0 tx=0x7efe48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe48011630 con 0x7efe581068d0 2024-03-26T04:39:46.517 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe48011790 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe58104880 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe48011a30 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe58104d20 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7efe48022070 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.846+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efe4801c450 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.847+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe580636b0 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.849+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7efe48011b90 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.921+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7efe58109450 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.923+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+338 (secure 0 0 0) 0x7efe48011de0 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.924+0000 7efe557fa700 1 -- 172.21.15.12:0/234107826 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7efe4801c6b0 con 0x7efe581068d0 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 msgr2=0x7efe58106150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 --2- 172.21.15.12:0/234107826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58106150 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efe58078af0 tx=0x7efe48004750 comp rx=0 tx=0).stop 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 shutdown_connections 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 --2- 172.21.15.12:0/234107826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efe581068d0 0x7efe58106150 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:46.518 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 >> 172.21.15.12:0/234107826 conn(0x7efe581002a0 msgr2=0x7efe581025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:46.519 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 shutdown_connections 2024-03-26T04:39:46.519 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:45.925+0000 7efe5e383700 1 -- 172.21.15.12:0/234107826 wait complete. 2024-03-26T04:39:46.519 INFO:teuthology.orchestra.run.smithi012.stdout:Generating new minimal ceph.conf... 2024-03-26T04:39:47.529 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:47 smithi012 ceph-mon[29493]: from='client.? 172.21.15.12:0/1477437392' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc76967f700 1 Processor -- start 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc76967f700 1 -- start start 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc76967f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764108c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc76967f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc764079710 con 0x7fc764106880 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc762ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764108c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc762ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764108c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52280/0 (socket says 172.21.15.12:52280) 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc762ffd700 1 -- 172.21.15.12:0/2417652796 learned_addr learned my addr 172.21.15.12:0/2417652796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.899+0000 7fc762ffd700 1 -- 172.21.15.12:0/2417652796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7641091b0 con 0x7fc764106880 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc762ffd700 1 --2- 172.21.15.12:0/2417652796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764108c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc7540098a0 tx=0x7fc754009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548394f1c342413e server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:47.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc761ffb700 1 -- 172.21.15.12:0/2417652796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc754004160 con 0x7fc764106880 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc761ffb700 1 -- 172.21.15.12:0/2417652796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc7540042c0 con 0x7fc764106880 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc761ffb700 1 -- 172.21.15.12:0/2417652796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7540118a0 con 0x7fc764106880 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 -- 172.21.15.12:0/2417652796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 msgr2=0x7fc764108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 --2- 172.21.15.12:0/2417652796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764108c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc7540098a0 tx=0x7fc754009bb0 comp rx=0 tx=0).stop 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 -- 172.21.15.12:0/2417652796 shutdown_connections 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 --2- 172.21.15.12:0/2417652796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764108c70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 -- 172.21.15.12:0/2417652796 >> 172.21.15.12:0/2417652796 conn(0x7fc7641002d0 msgr2=0x7fc7641026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 -- 172.21.15.12:0/2417652796 shutdown_connections 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.900+0000 7fc76967f700 1 -- 172.21.15.12:0/2417652796 wait complete. 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc76967f700 1 Processor -- start 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc76967f700 1 -- start start 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc76967f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764106100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc76967f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc764106640 con 0x7fc764106880 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc762ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764106100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc762ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764106100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52296/0 (socket says 172.21.15.12:52296) 2024-03-26T04:39:47.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.901+0000 7fc762ffd700 1 -- 172.21.15.12:0/1477437392 learned_addr learned my addr 172.21.15.12:0/1477437392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc762ffd700 1 -- 172.21.15.12:0/1477437392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754009550 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc762ffd700 1 --2- 172.21.15.12:0/1477437392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764106100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc754011890 tx=0x7fc754011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7540036a0 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc75401a430 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc764104830 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7540193c0 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc764104cd0 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc7540199d0 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.902+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc75400b720 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.903+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7640636b0 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.905+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc75402a060 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.976+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc76410a120 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.976+0000 7fc74bfff700 1 -- 172.21.15.12:0/1477437392 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fc75401a5a0 con 0x7fc764106880 2024-03-26T04:39:47.538 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 msgr2=0x7fc764106100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 --2- 172.21.15.12:0/1477437392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764106100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc754011890 tx=0x7fc754011970 comp rx=0 tx=0).stop 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 shutdown_connections 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 --2- 172.21.15.12:0/1477437392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc764106880 0x7fc764106100 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 >> 172.21.15.12:0/1477437392 conn(0x7fc7641002d0 msgr2=0x7fc764100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 shutdown_connections 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:46.977+0000 7fc76967f700 1 -- 172.21.15.12:0/1477437392 wait complete. 2024-03-26T04:39:47.539 INFO:teuthology.orchestra.run.smithi012.stdout:Restarting the monitor... 2024-03-26T04:39:47.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:47 smithi012 systemd[1]: Stopping Ceph mon.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:39:48.083 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:47 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a[29489]: 2024-03-26T04:39:47.806+0000 7ff7c23d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:39:48.083 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:47 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a[29489]: 2024-03-26T04:39:47.806+0000 7ff7c23d6700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T04:39:48.083 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:47 smithi012 podman[29739]: 2024-03-26 04:39:47.828198407 +0000 UTC m=+0.130556028 container died e9d04a958d619c3f9571f68fdaa1cc1f0256a68672426811803ef1ecea71109e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T04:39:48.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 podman[29739]: 2024-03-26 04:39:48.174011165 +0000 UTC m=+0.476368783 container cleanup e9d04a958d619c3f9571f68fdaa1cc1f0256a68672426811803ef1ecea71109e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T04:39:48.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 bash[29739]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:39:48.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 podman[29753]: 2024-03-26 04:39:48.420443423 +0000 UTC m=+0.592179993 container remove e9d04a958d619c3f9571f68fdaa1cc1f0256a68672426811803ef1ecea71109e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:39:48.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:39:48.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 systemd[1]: Stopped Ceph mon.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:39:48.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 systemd[1]: Starting Ceph mon.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:39:48.981 INFO:teuthology.orchestra.run.smithi012.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 podman[29844]: 2024-03-26 04:39:48.742657578 +0000 UTC m=+0.105463727 container create 86745ba2dacf6b5669dda0190eabc3655e67d8b2ecaf37a9d3305bd4c15761ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 podman[29844]: 2024-03-26 04:39:48.654665642 +0000 UTC m=+0.017471764 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 podman[29844]: 2024-03-26 04:39:48.911780994 +0000 UTC m=+0.274587108 container init 86745ba2dacf6b5669dda0190eabc3655e67d8b2ecaf37a9d3305bd4c15761ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 podman[29844]: 2024-03-26 04:39:48.915001517 +0000 UTC m=+0.277807635 container start 86745ba2dacf6b5669dda0190eabc3655e67d8b2ecaf37a9d3305bd4c15761ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: pidfile_write: ignore empty --pid-file 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: load: jerasure load: lrc 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Git sha 0 2024-03-26T04:39:49.017 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: DB SUMMARY 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: DB Session ID: GE4ORA2HG230Y8XQVDBB 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 86811 ; 2024-03-26T04:39:49.018 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.env: 0x55a7ef8b5ae0 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.info_log: 0x55a7f10cb920 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.statistics: (nil) 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.use_fsync: 0 2024-03-26T04:39:49.020 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.db_log_dir: 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.wal_dir: 2024-03-26T04:39:49.021 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.write_buffer_manager: 0x55a7f109e1e0 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:39:49.022 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.unordered_write: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.row_cache: None 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.wal_filter: None 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:39:49.023 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.wal_compression: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:39:49.024 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_open_files: -1 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:39:49.025 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Compression algorithms supported: 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kZSTD supported: 0 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:39:49.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.merge_operator: 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_filter: None 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a7f10cba20) 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_type: 0 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_index_type: 0 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_shortening: 1 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:39:49.027 INFO:journalctl@ceph.mon.a.smithi012.stdout: checksum: 4 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: no_block_cache: 0 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache: 0x55a7f10cf090 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_options: 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: capacity : 536870912 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_shard_bits : 4 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: strict_capacity_limit : 0 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_compressed: (nil) 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: persistent_cache: (nil) 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size: 4096 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size_deviation: 10 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_restart_interval: 16 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_block_restart_interval: 1 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: metadata_block_size: 4096 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: partition_filters: 0 2024-03-26T04:39:49.028 INFO:journalctl@ceph.mon.a.smithi012.stdout: use_delta_encoding: 1 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: filter_policy: bloomfilter 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: whole_key_filtering: 1 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: verify_compression: 0 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: format_version: 5 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_index_compression: 1 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_align: 0 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: max_auto_readahead_size: 262144 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: prepopulate_block_cache: 0 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression: NoCompression 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:39:49.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.num_levels: 7 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:39:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:39:49.031 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:39:49.032 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:39:49.033 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.ttl: 2592000 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:39:49.034 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dfd1d08d-c7d2-475b-a58d-8832f57f0d1a 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427988943770, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T04:39:49.035 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427988944677, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83233, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 262, "table_properties": {"data_size": 81405, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10592, "raw_average_key_size": 47, "raw_value_size": 75417, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 221, "num_filter_entries": 221, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711427988, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dfd1d08d-c7d2-475b-a58d-8832f57f0d1a", "db_session_id": "GE4ORA2HG230Y8XQVDBB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427988944763, "job": 1, "event": "recovery_finished"} 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a7f11c0000 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: DB pointer 0x55a7f11aa000 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** DB Stats ** 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:39:49.036 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:39:49.037 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative compaction: 0.00 GB write, 27.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval compaction: 0.00 GB write, 27.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache BinnedLRUCache@0x55a7f10cf090#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-26T04:39:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: starting mon.a rank 0 at public addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] at bind addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???) e1 preinit fsid c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).mds e1 new map 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout: e1 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout: legacy client fscid: -1 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout: No filesystems configured 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:39:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:39:49.040 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 ceph-mon[29863]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T04:39:49.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 bash[29844]: 86745ba2dacf6b5669dda0190eabc3655e67d8b2ecaf37a9d3305bd4c15761ad 2024-03-26T04:39:49.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:48 smithi012 systemd[1]: Started Ceph mon.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:39:49.041 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:49 smithi012 ceph-mon[29863]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:39:49.042 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:49 smithi012 ceph-mon[29863]: monmap e1: 1 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:39:49.042 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:49 smithi012 ceph-mon[29863]: fsmap 2024-03-26T04:39:49.042 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:49 smithi012 ceph-mon[29863]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:39:49.042 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:49 smithi012 ceph-mon[29863]: mgrmap e1: no daemons active 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.509+0000 7f9257633700 1 Processor -- start 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f9257633700 1 -- start start 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f9257633700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250108cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f9257633700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9250079690 con 0x7f92501068e0 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f92553cf700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250108cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f92553cf700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250108cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42858/0 (socket says 172.21.15.12:42858) 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f92553cf700 1 -- 172.21.15.12:0/620213348 learned_addr learned my addr 172.21.15.12:0/620213348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f92553cf700 1 -- 172.21.15.12:0/620213348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9250109210 con 0x7f92501068e0 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.510+0000 7f92553cf700 1 --2- 172.21.15.12:0/620213348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250108cd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f924c0098a0 tx=0x7f924c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5213ddc3e7ccb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9247fff700 1 -- 172.21.15.12:0/620213348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c004160 con 0x7f92501068e0 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9247fff700 1 -- 172.21.15.12:0/620213348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f924c0042c0 con 0x7f92501068e0 2024-03-26T04:39:50.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9247fff700 1 -- 172.21.15.12:0/620213348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c0118a0 con 0x7f92501068e0 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 -- 172.21.15.12:0/620213348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 msgr2=0x7f9250108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 --2- 172.21.15.12:0/620213348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250108cd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f924c0098a0 tx=0x7f924c009bb0 comp rx=0 tx=0).stop 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 -- 172.21.15.12:0/620213348 shutdown_connections 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 --2- 172.21.15.12:0/620213348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250108cd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 -- 172.21.15.12:0/620213348 >> 172.21.15.12:0/620213348 conn(0x7f92501002d0 msgr2=0x7f92501026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 -- 172.21.15.12:0/620213348 shutdown_connections 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.511+0000 7f9257633700 1 -- 172.21.15.12:0/620213348 wait complete. 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f9257633700 1 Processor -- start 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f9257633700 1 -- start start 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f9257633700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250106160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f9257633700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92501066a0 con 0x7f92501068e0 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f92553cf700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250106160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f92553cf700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250106160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42860/0 (socket says 172.21.15.12:42860) 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f92553cf700 1 -- 172.21.15.12:0/2435224176 learned_addr learned my addr 172.21.15.12:0/2435224176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:50.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.512+0000 7f92553cf700 1 -- 172.21.15.12:0/2435224176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c009550 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f92553cf700 1 --2- 172.21.15.12:0/2435224176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250106160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f924c0046c0 tx=0x7f924c011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c0036a0 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f924c01a430 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9250104890 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f924c0193c0 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9250104d30 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f924c019980 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.513+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f924c024730 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.514+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92500636b0 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.516+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f924c01a930 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.585+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f925010a0d0 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.587+0000 7f92467fc700 1 -- 172.21.15.12:0/2435224176 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f924c003800 con 0x7f92501068e0 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 msgr2=0x7f9250106160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:50.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 --2- 172.21.15.12:0/2435224176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250106160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f924c0046c0 tx=0x7f924c011eb0 comp rx=0 tx=0).stop 2024-03-26T04:39:50.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 shutdown_connections 2024-03-26T04:39:50.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 --2- 172.21.15.12:0/2435224176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f92501068e0 0x7f9250106160 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:50.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 >> 172.21.15.12:0/2435224176 conn(0x7f92501002d0 msgr2=0x7f9250100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:50.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 shutdown_connections 2024-03-26T04:39:50.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:49.588+0000 7f9257633700 1 -- 172.21.15.12:0/2435224176 wait complete. 2024-03-26T04:39:50.138 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T04:39:50.159 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:39:50.160 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mgr... 2024-03-26T04:39:50.160 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T04:39:50.161 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T04:39:50.410 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:39:50.410 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T04:39:50.563 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715.target.wants/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@.service. 2024-03-26T04:39:50.796 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2435224176' entity='client.admin' 2024-03-26T04:39:51.064 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:51 smithi012 podman[30079]: 2024-03-26 04:39:51.009259497 +0000 UTC m=+0.327917692 container start d473c37e3fccca5790f4966958c04a24bd12ebdf73abf8a544dc24cabb642a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T04:39:51.078 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:39:51.078 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-26T04:39:51.085 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:39:51.085 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-26T04:39:51.085 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-26T04:39:51.085 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:39:51.093 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:39:51.093 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-26T04:39:51.100 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:39:51.100 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-26T04:39:51.100 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-26T04:39:51.100 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T04:39:51.100 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr to start... 2024-03-26T04:39:51.101 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr... 2024-03-26T04:39:51.330 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:51 smithi012 bash[30079]: d473c37e3fccca5790f4966958c04a24bd12ebdf73abf8a544dc24cabb642a60 2024-03-26T04:39:51.330 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:51 smithi012 systemd[1]: Started Ceph mgr.y for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:39:51.583 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:51 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:51.571+0000 7f5b054ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:39:51.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/938876600' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:39:51.862 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:51 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:51.671+0000 7f5b054ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "c3b00422-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:39:52.093 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:39:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:39:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:39:52.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:39:52.106 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:39:52.106 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:39:52.106 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:39:52.106 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.106 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:39:52.107 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:39:44.325567+0000", 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:52.108 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e9c51700 1 Processor -- start 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e9c51700 1 -- start start 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e9c51700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e4079b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e9c51700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e407a130 con 0x7f01e407b700 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e4079b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e4079b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42896/0 (socket says 172.21.15.12:42896) 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.502+0000 7f01e37fe700 1 -- 172.21.15.12:0/818371827 learned_addr learned my addr 172.21.15.12:0/818371827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.503+0000 7f01e37fe700 1 -- 172.21.15.12:0/818371827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e407a270 con 0x7f01e407b700 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.503+0000 7f01e37fe700 1 --2- 172.21.15.12:0/818371827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e4079b60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0098a0 tx=0x7f01cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457445b1934747a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.503+0000 7f01e27fc700 1 -- 172.21.15.12:0/818371827 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01cc004160 con 0x7f01e407b700 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.503+0000 7f01e27fc700 1 -- 172.21.15.12:0/818371827 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f01cc0042c0 con 0x7f01e407b700 2024-03-26T04:39:52.109 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.503+0000 7f01e27fc700 1 -- 172.21.15.12:0/818371827 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01cc0118a0 con 0x7f01e407b700 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 -- 172.21.15.12:0/818371827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 msgr2=0x7f01e4079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 --2- 172.21.15.12:0/818371827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e4079b60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0098a0 tx=0x7f01cc009bb0 comp rx=0 tx=0).stop 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 -- 172.21.15.12:0/818371827 shutdown_connections 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 --2- 172.21.15.12:0/818371827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e4079b60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 -- 172.21.15.12:0/818371827 >> 172.21.15.12:0/818371827 conn(0x7f01e41013b0 msgr2=0x7f01e41037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 -- 172.21.15.12:0/818371827 shutdown_connections 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 -- 172.21.15.12:0/818371827 wait complete. 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.504+0000 7f01e9c51700 1 Processor -- start 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e9c51700 1 -- start start 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e9c51700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e419bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e9c51700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e419c1d0 con 0x7f01e407b700 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e419bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:52.110 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e419bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42900/0 (socket says 172.21.15.12:42900) 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e37fe700 1 -- 172.21.15.12:0/938876600 learned_addr learned my addr 172.21.15.12:0/938876600 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e37fe700 1 -- 172.21.15.12:0/938876600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc009550 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e37fe700 1 --2- 172.21.15.12:0/938876600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e419bc90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01cc003da0 tx=0x7f01cc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01cc011930 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f01cc003b00 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.505+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e419c3d0 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.506+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01cc01ad50 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.506+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e419c870 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.506+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f01cc02d7d0 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.506+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f01cc019700 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.506+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e410c120 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.509+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f01cc01a630 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.583+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f01e419f250 con 0x7f01e407b700 2024-03-26T04:39:52.111 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.583+0000 7f01e0ff9700 1 -- 172.21.15.12:0/938876600 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f01cc0036a0 con 0x7f01e407b700 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 msgr2=0x7f01e419bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 --2- 172.21.15.12:0/938876600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e419bc90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01cc003da0 tx=0x7f01cc011e90 comp rx=0 tx=0).stop 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 shutdown_connections 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 --2- 172.21.15.12:0/938876600 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f01e407b700 0x7f01e419bc90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 >> 172.21.15.12:0/938876600 conn(0x7f01e41013b0 msgr2=0x7f01e4101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 shutdown_connections 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:51.585+0000 7f01e9c51700 1 -- 172.21.15.12:0/938876600 wait complete. 2024-03-26T04:39:52.112 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (1/15)... 2024-03-26T04:39:52.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:52.333+0000 7f5b054ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:39:52.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:52.427+0000 7f5b054ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:39:52.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:52.569+0000 7f5b054ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:39:53.017 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:52.666+0000 7f5b054ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:39:53.017 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:52.751+0000 7f5b054ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:39:53.017 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:53.017+0000 7f5b054ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:39:53.311 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:53.128+0000 7f5b054ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:39:53.312 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:53.215+0000 7f5b054ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:39:53.312 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:53.311+0000 7f5b054ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:39:53.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:53.488+0000 7f5b054ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:39:54.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1690488209' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:39:54.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:54.495+0000 7f5b054ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:39:54.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:54.636+0000 7f5b054ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:39:55.236 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:54.978+0000 7f5b054ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:39:55.236 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:55.069+0000 7f5b054ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:39:55.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:55.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:39:55.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "c3b00422-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:39:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:39:55.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:39:55.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:55.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:39:55.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:39:44.325567+0000", 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.535+0000 7fa7dff95700 1 Processor -- start 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7dff95700 1 -- start start 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7dff95700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8105100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7dff95700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d81056d0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7ddd31700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8105100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:55.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7ddd31700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8105100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42906/0 (socket says 172.21.15.12:42906) 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7ddd31700 1 -- 172.21.15.12:0/1715978590 learned_addr learned my addr 172.21.15.12:0/1715978590 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7ddd31700 1 -- 172.21.15.12:0/1715978590 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8105ee0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.536+0000 7fa7ddd31700 1 --2- 172.21.15.12:0/1715978590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8105100 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40098a0 tx=0x7fa7d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb45055393b0a5a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dcd2f700 1 -- 172.21.15.12:0/1715978590 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7d4004160 con 0x7fa7d8104ce0 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dcd2f700 1 -- 172.21.15.12:0/1715978590 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fa7d40042c0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dcd2f700 1 -- 172.21.15.12:0/1715978590 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7d40118a0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 -- 172.21.15.12:0/1715978590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 msgr2=0x7fa7d8105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 --2- 172.21.15.12:0/1715978590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8105100 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40098a0 tx=0x7fa7d4009bb0 comp rx=0 tx=0).stop 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 -- 172.21.15.12:0/1715978590 shutdown_connections 2024-03-26T04:39:55.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 --2- 172.21.15.12:0/1715978590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8105100 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 -- 172.21.15.12:0/1715978590 >> 172.21.15.12:0/1715978590 conn(0x7fa7d8100260 msgr2=0x7fa7d81026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 -- 172.21.15.12:0/1715978590 shutdown_connections 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.537+0000 7fa7dff95700 1 -- 172.21.15.12:0/1715978590 wait complete. 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7dff95700 1 Processor -- start 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7dff95700 1 -- start start 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7dff95700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7dff95700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d819a030 con 0x7fa7d8104ce0 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7ddd31700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7ddd31700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42916/0 (socket says 172.21.15.12:42916) 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.538+0000 7fa7ddd31700 1 -- 172.21.15.12:0/1690488209 learned_addr learned my addr 172.21.15.12:0/1690488209 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:55.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7ddd31700 1 -- 172.21.15.12:0/1690488209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d4009550 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7ddd31700 1 --2- 172.21.15.12:0/1690488209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8199af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4003f10 tx=0x7fa7d4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7d40036a0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fa7d401a430 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d819a230 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7d4024410 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d819a6d0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa7d4024a10 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.539+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa7d40193c0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.540+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d8193c00 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.542+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa7d402de40 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.616+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa7d80636b0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.617+0000 7fa7caffd700 1 -- 172.21.15.12:0/1690488209 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fa7d401a5a0 con 0x7fa7d8104ce0 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 msgr2=0x7fa7d8199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 --2- 172.21.15.12:0/1690488209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8199af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4003f10 tx=0x7fa7d4011940 comp rx=0 tx=0).stop 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 shutdown_connections 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 --2- 172.21.15.12:0/1690488209 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7d8104ce0 0x7fa7d8199af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 >> 172.21.15.12:0/1690488209 conn(0x7fa7d8100260 msgr2=0x7fa7d81011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 shutdown_connections 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:54.618+0000 7fa7dff95700 1 -- 172.21.15.12:0/1690488209 wait complete. 2024-03-26T04:39:55.261 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (2/15)... 2024-03-26T04:39:55.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:55.271+0000 7f5b054ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:39:55.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:55.459+0000 7f5b054ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:39:56.008 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:55.747+0000 7f5b054ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:39:56.008 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:56.007+0000 7f5b054ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:39:56.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:56.248+0000 7f5b054ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:39:57.368 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:57.069+0000 7f5b054ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:39:57.987 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3338560228' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:39:57.987 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:57.659+0000 7f5b054ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:39:57.987 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:57.762+0000 7f5b054ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:39:57.987 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:39:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:39:57.986+0000 7f5b054ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "c3b00422-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:39:58.324 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.325 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:39:58.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:58.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:39:58.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:39:58.326 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:39:58.334 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:39:58.335 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:39:44.325567+0000", 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:39:58.336 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.673+0000 7fd0832a9700 1 Processor -- start 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.673+0000 7fd0832a9700 1 -- start start 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.673+0000 7fd0832a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c106200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.673+0000 7fd0832a9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1067d0 con 0x7fd07c105de0 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd081045700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c106200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd081045700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c106200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42926/0 (socket says 172.21.15.12:42926) 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd081045700 1 -- 172.21.15.12:0/3180661695 learned_addr learned my addr 172.21.15.12:0/3180661695 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd081045700 1 -- 172.21.15.12:0/3180661695 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c106fe0 con 0x7fd07c105de0 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd081045700 1 --2- 172.21.15.12:0/3180661695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c106200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0098a0 tx=0x7fd06c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9188f28d25ecc07 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:58.337 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd073fff700 1 -- 172.21.15.12:0/3180661695 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd06c004160 con 0x7fd07c105de0 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd073fff700 1 -- 172.21.15.12:0/3180661695 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd06c0042c0 con 0x7fd07c105de0 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.674+0000 7fd073fff700 1 -- 172.21.15.12:0/3180661695 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd06c0118a0 con 0x7fd07c105de0 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 -- 172.21.15.12:0/3180661695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 msgr2=0x7fd07c106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 --2- 172.21.15.12:0/3180661695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c106200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0098a0 tx=0x7fd06c009bb0 comp rx=0 tx=0).stop 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 -- 172.21.15.12:0/3180661695 shutdown_connections 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 --2- 172.21.15.12:0/3180661695 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c106200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 -- 172.21.15.12:0/3180661695 >> 172.21.15.12:0/3180661695 conn(0x7fd07c1013c0 msgr2=0x7fd07c1037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 -- 172.21.15.12:0/3180661695 shutdown_connections 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.675+0000 7fd0832a9700 1 -- 172.21.15.12:0/3180661695 wait complete. 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd0832a9700 1 Processor -- start 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd0832a9700 1 -- start start 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd0832a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd0832a9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c19a030 con 0x7fd07c105de0 2024-03-26T04:39:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd081045700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd081045700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42934/0 (socket says 172.21.15.12:42934) 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd081045700 1 -- 172.21.15.12:0/3338560228 learned_addr learned my addr 172.21.15.12:0/3338560228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd081045700 1 -- 172.21.15.12:0/3338560228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c009550 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.676+0000 7fd081045700 1 --2- 172.21.15.12:0/3338560228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c199af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd06c006af0 tx=0x7fd06c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd06c0036a0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd06c01a430 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c19a230 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd06c0193c0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c19a6d0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd06c019980 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd06c02c3a0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.677+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c0636b0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.680+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd06c019dc0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.754+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd07c10a1c0 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.754+0000 7fd0727fc700 1 -- 172.21.15.12:0/3338560228 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd06c031030 con 0x7fd07c105de0 2024-03-26T04:39:58.339 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 msgr2=0x7fd07c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 --2- 172.21.15.12:0/3338560228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c199af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd06c006af0 tx=0x7fd06c0119b0 comp rx=0 tx=0).stop 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 shutdown_connections 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 --2- 172.21.15.12:0/3338560228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd07c105de0 0x7fd07c199af0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 >> 172.21.15.12:0/3338560228 conn(0x7fd07c1013c0 msgr2=0x7fd07c101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 shutdown_connections 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:39:57.756+0000 7fd0832a9700 1 -- 172.21.15.12:0/3338560228 wait complete. 2024-03-26T04:39:58.340 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (3/15)... 2024-03-26T04:39:59.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: Activating manager daemon y 2024-03-26T04:39:59.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: mgrmap e2: y(active, starting, since 0.00297968s) 2024-03-26T04:39:59.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: Manager daemon y is now available 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' 2024-03-26T04:39:59.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:39:58 smithi012 ceph-mon[29863]: from='mgr.14100 172.21.15.12:0/2747229574' entity='mgr.y' 2024-03-26T04:40:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:00 smithi012 ceph-mon[29863]: mgrmap e3: y(active, since 1.00747s) 2024-03-26T04:40:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:02 smithi012 ceph-mon[29863]: mgrmap e4: y(active, since 3s) 2024-03-26T04:40:04.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1068428658' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:40:08.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:40:08.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:40:08.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "c3b00422-eb2a-11ee-95d0-87774f69a715", 2024-03-26T04:40:08.723 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:40:08.724 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:40:08.725 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:40:08.734 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:40:08.735 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:39:44.325567+0000", 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd03f64700 1 Processor -- start 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd03f64700 1 -- start start 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd03f64700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc079b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd03f64700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc07a130 con 0x7fccfc07b700 2024-03-26T04:40:08.736 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd01d00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc079b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd01d00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc079b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59602/0 (socket says 172.21.15.12:59602) 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.854+0000 7fcd01d00700 1 -- 172.21.15.12:0/4294096753 learned_addr learned my addr 172.21.15.12:0/4294096753 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd01d00700 1 -- 172.21.15.12:0/4294096753 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc07a270 con 0x7fccfc07b700 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd01d00700 1 --2- 172.21.15.12:0/4294096753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc079b60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fccf80098a0 tx=0x7fccf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6738d91fed8ea79 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd00cfe700 1 -- 172.21.15.12:0/4294096753 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccf8004160 con 0x7fccfc07b700 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd00cfe700 1 -- 172.21.15.12:0/4294096753 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fccf80042c0 con 0x7fccfc07b700 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd00cfe700 1 -- 172.21.15.12:0/4294096753 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccf80118a0 con 0x7fccfc07b700 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd03f64700 1 -- 172.21.15.12:0/4294096753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 msgr2=0x7fccfc079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd03f64700 1 --2- 172.21.15.12:0/4294096753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc079b60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fccf80098a0 tx=0x7fccf8009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd03f64700 1 -- 172.21.15.12:0/4294096753 shutdown_connections 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd03f64700 1 --2- 172.21.15.12:0/4294096753 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc079b60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.855+0000 7fcd03f64700 1 -- 172.21.15.12:0/4294096753 >> 172.21.15.12:0/4294096753 conn(0x7fccfc1013b0 msgr2=0x7fccfc1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.856+0000 7fcd03f64700 1 -- 172.21.15.12:0/4294096753 shutdown_connections 2024-03-26T04:40:08.737 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.856+0000 7fcd03f64700 1 -- 172.21.15.12:0/4294096753 wait complete. 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.856+0000 7fcd03f64700 1 Processor -- start 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.856+0000 7fcd03f64700 1 -- start start 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.856+0000 7fcd03f64700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.856+0000 7fcd03f64700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc19c1f0 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd01d00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd01d00700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59610/0 (socket says 172.21.15.12:59610) 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd01d00700 1 -- 172.21.15.12:0/1068428658 learned_addr learned my addr 172.21.15.12:0/1068428658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd01d00700 1 -- 172.21.15.12:0/1068428658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf8009550 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd01d00700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc19bcb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccf8006af0 tx=0x7fccf80119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccf80036a0 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fccf801a430 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc19c3f0 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fccf80193c0 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.857+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc19c890 con 0x7fccfc07b700 2024-03-26T04:40:08.738 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.858+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 45917+0+0 (secure 0 0 0) 0x7fccf8019520 con 0x7fccfc07b700 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.858+0000 7fccf2ffd700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fcce8038ee0 0x7fcce803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.858+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fccf804cb30 con 0x7fccfc07b700 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.858+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc10c100 con 0x7fccfc07b700 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.858+0000 7fcd014ff700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fcce8038ee0 0x7fcce803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.859+0000 7fcd014ff700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fcce8038ee0 0x7fcce803b3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fccec006fd0 tx=0x7fccec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:03.864+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fccf802c3a0 con 0x7fccfc07b700 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.153+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fccfc19f270 con 0x7fccfc07b700 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.154+0000 7fccf2ffd700 1 -- 172.21.15.12:0/1068428658 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fccf8010380 con 0x7fccfc07b700 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fcce8038ee0 msgr2=0x7fcce803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fcce8038ee0 0x7fcce803b3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fccec006fd0 tx=0x7fccec006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 msgr2=0x7fccfc19bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc19bcb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccf8006af0 tx=0x7fccf80119b0 comp rx=0 tx=0).stop 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 shutdown_connections 2024-03-26T04:40:08.739 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fcce8038ee0 0x7fcce803b3a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:08.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 --2- 172.21.15.12:0/1068428658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fccfc07b700 0x7fccfc19bcb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:08.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 >> 172.21.15.12:0/1068428658 conn(0x7fccfc1013b0 msgr2=0x7fccfc101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:08.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 shutdown_connections 2024-03-26T04:40:08.740 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:04.156+0000 7fcd03f64700 1 -- 172.21.15.12:0/1068428658 wait complete. 2024-03-26T04:40:08.740 INFO:teuthology.orchestra.run.smithi012.stdout:mgr is available 2024-03-26T04:40:09.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3765146055' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:40:09.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:40:09.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:40:09.900 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58ba223700 1 Processor -- start 2024-03-26T04:40:09.901 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58ba223700 1 -- start start 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58ba223700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b41048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58ba223700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4104e80 con 0x7f58b4104490 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58b37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b41048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58b37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b41048b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49966/0 (socket says 172.21.15.12:49966) 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58b37fe700 1 -- 172.21.15.12:0/20152684 learned_addr learned my addr 172.21.15.12:0/20152684 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.125+0000 7f58b37fe700 1 -- 172.21.15.12:0/20152684 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4104fc0 con 0x7f58b4104490 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58b37fe700 1 --2- 172.21.15.12:0/20152684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b41048b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f58a40098a0 tx=0x7f58a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd03fe99ae9a303 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58b27fc700 1 -- 172.21.15.12:0/20152684 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58a4004160 con 0x7f58b4104490 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58b27fc700 1 -- 172.21.15.12:0/20152684 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f58a40042c0 con 0x7f58b4104490 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58b27fc700 1 -- 172.21.15.12:0/20152684 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58a40118a0 con 0x7f58b4104490 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 -- 172.21.15.12:0/20152684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 msgr2=0x7f58b41048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 --2- 172.21.15.12:0/20152684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b41048b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f58a40098a0 tx=0x7f58a4009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:09.902 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 -- 172.21.15.12:0/20152684 shutdown_connections 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 --2- 172.21.15.12:0/20152684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b41048b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 -- 172.21.15.12:0/20152684 >> 172.21.15.12:0/20152684 conn(0x7f58b40ff570 msgr2=0x7f58b41019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 -- 172.21.15.12:0/20152684 shutdown_connections 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.126+0000 7f58ba223700 1 -- 172.21.15.12:0/20152684 wait complete. 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58ba223700 1 Processor -- start 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58ba223700 1 -- start start 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58ba223700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58ba223700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4199fd0 con 0x7f58b4104490 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58b37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58b37fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b4199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49976/0 (socket says 172.21.15.12:49976) 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.127+0000 7f58b37fe700 1 -- 172.21.15.12:0/3765146055 learned_addr learned my addr 172.21.15.12:0/3765146055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58b37fe700 1 -- 172.21.15.12:0/3765146055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4009550 con 0x7f58b4104490 2024-03-26T04:40:09.903 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58b37fe700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b4199a90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f58a4003f10 tx=0x7f58a4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58a40036a0 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f58a401a430 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b419a1d0 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58a4024410 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.128+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b419a670 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.129+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 45917+0+0 (secure 0 0 0) 0x7f58a402d3f0 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.129+0000 7f58b0ff9700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f589c038ee0 0x7f589c03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.129+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f58a404cb90 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.129+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b4193680 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.129+0000 7f58b2ffd700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f589c038ee0 0x7f589c03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.130+0000 7f58b2ffd700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f589c038ee0 0x7f589c03b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58a8006fd0 tx=0x7f58a8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.134+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f58a4010740 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.337+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f58b4105a80 con 0x7f58b4104490 2024-03-26T04:40:09.904 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.338+0000 7f58b0ff9700 1 -- 172.21.15.12:0/3765146055 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f58a4010380 con 0x7f58b4104490 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f589c038ee0 msgr2=0x7f589c03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f589c038ee0 0x7f589c03b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58a8006fd0 tx=0x7f58a8006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 msgr2=0x7f58b4199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b4199a90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f58a4003f10 tx=0x7f58a4011940 comp rx=0 tx=0).stop 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 shutdown_connections 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f589c038ee0 0x7f589c03b3a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 --2- 172.21.15.12:0/3765146055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f58b4104490 0x7f58b4199a90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 >> 172.21.15.12:0/3765146055 conn(0x7f58b40ff570 msgr2=0x7f58b41907c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 shutdown_connections 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:09.340+0000 7f58ba223700 1 -- 172.21.15.12:0/3765146055 wait complete. 2024-03-26T04:40:09.905 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling cephadm module... 2024-03-26T04:40:10.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3668433988' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T04:40:10.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:10 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ignoring --setuser ceph since I am not root 2024-03-26T04:40:10.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:10 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ignoring --setgroup ceph since I am not root 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc7a11fe700 1 Processor -- start 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc7a11fe700 1 -- start start 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc7a11fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c106230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc7a11fe700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c106770 con 0x7fc79c105e10 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc79ad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc79ad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49982/0 (socket says 172.21.15.12:49982) 2024-03-26T04:40:11.123 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc79ad9d700 1 -- 172.21.15.12:0/4201530063 learned_addr learned my addr 172.21.15.12:0/4201530063 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.292+0000 7fc79ad9d700 1 -- 172.21.15.12:0/4201530063 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c1068b0 con 0x7fc79c105e10 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc79ad9d700 1 --2- 172.21.15.12:0/4201530063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c106230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0098a0 tx=0x7fc78c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7581ff68fd485021 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc799d9b700 1 -- 172.21.15.12:0/4201530063 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc78c004160 con 0x7fc79c105e10 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc799d9b700 1 -- 172.21.15.12:0/4201530063 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc78c0042c0 con 0x7fc79c105e10 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc799d9b700 1 -- 172.21.15.12:0/4201530063 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc78c0118a0 con 0x7fc79c105e10 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 -- 172.21.15.12:0/4201530063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 msgr2=0x7fc79c106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 --2- 172.21.15.12:0/4201530063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c106230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0098a0 tx=0x7fc78c009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 -- 172.21.15.12:0/4201530063 shutdown_connections 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 --2- 172.21.15.12:0/4201530063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c106230 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 -- 172.21.15.12:0/4201530063 >> 172.21.15.12:0/4201530063 conn(0x7fc79c1013b0 msgr2=0x7fc79c1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 -- 172.21.15.12:0/4201530063 shutdown_connections 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.293+0000 7fc7a11fe700 1 -- 172.21.15.12:0/4201530063 wait complete. 2024-03-26T04:40:11.124 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.294+0000 7fc7a11fe700 1 Processor -- start 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.294+0000 7fc7a11fe700 1 -- start start 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.294+0000 7fc7a11fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.294+0000 7fc7a11fe700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c199f80 con 0x7fc79c105e10 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc79ad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc79ad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49994/0 (socket says 172.21.15.12:49994) 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc79ad9d700 1 -- 172.21.15.12:0/3668433988 learned_addr learned my addr 172.21.15.12:0/3668433988 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc79ad9d700 1 -- 172.21.15.12:0/3668433988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c009550 con 0x7fc79c105e10 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc79ad9d700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c199a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc78c003da0 tx=0x7fc78c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc78c011930 con 0x7fc79c105e10 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c19a180 con 0x7fc79c105e10 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc78c003b00 con 0x7fc79c105e10 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc78c01ad50 con 0x7fc79c105e10 2024-03-26T04:40:11.125 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.295+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c19a620 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.296+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) v1 ==== 45917+0+0 (secure 0 0 0) 0x7fc78c02d440 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.296+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79c0636b0 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.296+0000 7fc783fff700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc784038e40 0x7fc78403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.296+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc78c04d000 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.297+0000 7fc79a59c700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc784038e40 0x7fc78403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.297+0000 7fc79a59c700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc784038e40 0x7fc78403b300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc790006fd0 tx=0x7fc790006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.301+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc78c018710 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.561+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc79c107b80 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.612+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc78c00b3e0 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.614+0000 7fc783fff700 1 -- 172.21.15.12:0/3668433988 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 5) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fc78c01a630 con 0x7fc79c105e10 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc784038e40 msgr2=0x7fc78403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:11.126 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc784038e40 0x7fc78403b300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc790006fd0 tx=0x7fc790006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 msgr2=0x7fc79c199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c199a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc78c003da0 tx=0x7fc78c011e90 comp rx=0 tx=0).stop 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 shutdown_connections 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc784038e40 0x7fc78403b300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 --2- 172.21.15.12:0/3668433988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc79c105e10 0x7fc79c199a40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 >> 172.21.15.12:0/3668433988 conn(0x7fc79c1013b0 msgr2=0x7fc79c101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 shutdown_connections 2024-03-26T04:40:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:10.616+0000 7fc7a11fe700 1 -- 172.21.15.12:0/3668433988 wait complete. 2024-03-26T04:40:11.617 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:11 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:11.183+0000 7f524c2f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:40:11.618 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:11 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:11.279+0000 7f524c2f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:40:11.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3668433988' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T04:40:11.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:11 smithi012 ceph-mon[29863]: mgrmap e5: y(active, since 12s) 2024-03-26T04:40:12.177 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:11 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:11.936+0000 7f524c2f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:40:12.177 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.031+0000 7f524c2f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:40:12.177 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.176+0000 7f524c2f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:40:12.451 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.279+0000 7f524c2f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:40:12.451 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.369+0000 7f524c2f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:40:12.766 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2987433784' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:40:12.766 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.650+0000 7f524c2f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:40:12.766 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.765+0000 7f524c2f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:13.134 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.856+0000 7f524c2f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:40:13.134 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:12 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:12.955+0000 7f524c2f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:40:13.134 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:13 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:13.133+0000 7f524c2f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:40:13.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:40:13.249 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T04:40:13.249 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:40:13.249 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.336+0000 7f980e9d9700 1 Processor -- start 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.336+0000 7f980e9d9700 1 -- start start 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.336+0000 7f980e9d9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808106210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.336+0000 7f980e9d9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98081067e0 con 0x7f9808105df0 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9807fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9807fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50018/0 (socket says 172.21.15.12:50018) 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9807fff700 1 -- 172.21.15.12:0/2360500340 learned_addr learned my addr 172.21.15.12:0/2360500340 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9807fff700 1 -- 172.21.15.12:0/2360500340 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9808106ff0 con 0x7f9808105df0 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9807fff700 1 --2- 172.21.15.12:0/2360500340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808106210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97f80098a0 tx=0x7f97f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f090f1b7b59c186 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9806ffd700 1 -- 172.21.15.12:0/2360500340 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97f8004160 con 0x7f9808105df0 2024-03-26T04:40:13.250 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9806ffd700 1 -- 172.21.15.12:0/2360500340 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f97f80042c0 con 0x7f9808105df0 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.337+0000 7f9806ffd700 1 -- 172.21.15.12:0/2360500340 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97f80118a0 con 0x7f9808105df0 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 -- 172.21.15.12:0/2360500340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 msgr2=0x7f9808106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 --2- 172.21.15.12:0/2360500340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808106210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97f80098a0 tx=0x7f97f8009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 -- 172.21.15.12:0/2360500340 shutdown_connections 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 --2- 172.21.15.12:0/2360500340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808106210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 -- 172.21.15.12:0/2360500340 >> 172.21.15.12:0/2360500340 conn(0x7f98081013f0 msgr2=0x7f9808103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 -- 172.21.15.12:0/2360500340 shutdown_connections 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.338+0000 7f980e9d9700 1 -- 172.21.15.12:0/2360500340 wait complete. 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.339+0000 7f980e9d9700 1 Processor -- start 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.339+0000 7f980e9d9700 1 -- start start 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.339+0000 7f980e9d9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.339+0000 7f980e9d9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9808199ff0 con 0x7f9808105df0 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.339+0000 7f9807fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:13.251 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f9807fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50026/0 (socket says 172.21.15.12:50026) 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f9807fff700 1 -- 172.21.15.12:0/2987433784 learned_addr learned my addr 172.21.15.12:0/2987433784 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f9807fff700 1 -- 172.21.15.12:0/2987433784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8009550 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f9807fff700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808199ab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f8003da0 tx=0x7f97f8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97f8011930 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f97f8003b00 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980819a1f0 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97f801ad50 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.340+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980819a690 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.341+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f97f802d440 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.342+0000 7f98057fa700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 0x7f97f003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.342+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f97f804ce70 con 0x7f9808105df0 2024-03-26T04:40:13.252 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.342+0000 7f98077fe700 1 -- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 msgr2=0x7f97f003b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.342+0000 7f98077fe700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 0x7f97f003b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.342+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98080636b0 con 0x7f9808105df0 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.347+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f97f802d740 con 0x7f9808105df0 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.543+0000 7f98077fe700 1 -- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 msgr2=0x7f97f003b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.543+0000 7f98077fe700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 0x7f97f003b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.610+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f980819d050 con 0x7f9808105df0 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.611+0000 7f98057fa700 1 -- 172.21.15.12:0/2987433784 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f97f8018860 con 0x7f9808105df0 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 msgr2=0x7f97f003b350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 0x7f97f003b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 msgr2=0x7f9808199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808199ab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f8003da0 tx=0x7f97f8011e90 comp rx=0 tx=0).stop 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 shutdown_connections 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7f97f0038e90 0x7f97f003b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:13.253 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 --2- 172.21.15.12:0/2987433784 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9808105df0 0x7f9808199ab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:13.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 >> 172.21.15.12:0/2987433784 conn(0x7f98081013f0 msgr2=0x7f9808101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:13.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 shutdown_connections 2024-03-26T04:40:13.254 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:12.614+0000 7f980e9d9700 1 -- 172.21.15.12:0/2987433784 wait complete. 2024-03-26T04:40:13.254 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-03-26T04:40:13.254 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 5... 2024-03-26T04:40:14.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:14 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:14.166+0000 7f524c2f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:40:14.655 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:14 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:14.299+0000 7f524c2f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:40:14.947 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:14 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:14.657+0000 7f524c2f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:40:14.948 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:14 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:14.748+0000 7f524c2f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:40:14.948 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:14 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:14.947+0000 7f524c2f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:15.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:15.134+0000 7f524c2f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:40:15.683 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:15.422+0000 7f524c2f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:15.684 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:15.683+0000 7f524c2f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:16.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:15.923+0000 7f524c2f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:40:17.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:16.742+0000 7f524c2f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:40:17.638 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:17.319+0000 7f524c2f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:40:17.638 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:17.419+0000 7f524c2f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:40:17.638 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:17.637+0000 7f524c2f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: Active manager daemon y restarted 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: Activating manager daemon y 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: mgrmap e6: y(active, starting, since 0.00331377s) 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:40:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:40:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:40:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:40:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: Manager daemon y is now available 2024-03-26T04:40:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:17 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: Found migration_current of "None". Setting to last migration. 2024-03-26T04:40:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:40:18.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: mgrmap e7: y(active, since 1.00595s) 2024-03-26T04:40:18.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:18.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:18 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:19.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:40:19.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T04:40:19.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.677+0000 7fc0668b7700 1 Processor -- start 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.677+0000 7fc0668b7700 1 -- start start 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.677+0000 7fc0668b7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060108c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.677+0000 7fc0668b7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060079710 con 0x7fc060106850 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50038/0 (socket says 172.21.15.12:50038) 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05ffff700 1 -- 172.21.15.12:0/3955080572 learned_addr learned my addr 172.21.15.12:0/3955080572 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05ffff700 1 -- 172.21.15.12:0/3955080572 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc060109180 con 0x7fc060106850 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05ffff700 1 --2- 172.21.15.12:0/3955080572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060108c40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0480098a0 tx=0x7fc048009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31332f1b69e76637 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05effd700 1 -- 172.21.15.12:0/3955080572 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc048004160 con 0x7fc060106850 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05effd700 1 -- 172.21.15.12:0/3955080572 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc0480042c0 con 0x7fc060106850 2024-03-26T04:40:19.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.678+0000 7fc05effd700 1 -- 172.21.15.12:0/3955080572 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0480118a0 con 0x7fc060106850 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 -- 172.21.15.12:0/3955080572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 msgr2=0x7fc060108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3955080572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060108c40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc0480098a0 tx=0x7fc048009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 -- 172.21.15.12:0/3955080572 shutdown_connections 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3955080572 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060108c40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 -- 172.21.15.12:0/3955080572 >> 172.21.15.12:0/3955080572 conn(0x7fc0601002e0 msgr2=0x7fc060102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 -- 172.21.15.12:0/3955080572 shutdown_connections 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 -- 172.21.15.12:0/3955080572 wait complete. 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.679+0000 7fc0668b7700 1 Processor -- start 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc0668b7700 1 -- start start 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc0668b7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc0668b7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060199fe0 con 0x7fc060106850 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc05ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50040/0 (socket says 172.21.15.12:50040) 2024-03-26T04:40:19.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc05ffff700 1 -- 172.21.15.12:0/3439662715 learned_addr learned my addr 172.21.15.12:0/3439662715 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.680+0000 7fc05ffff700 1 -- 172.21.15.12:0/3439662715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc048009550 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.681+0000 7fc05ffff700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060199aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc0480046c0 tx=0x7fc048011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.681+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0480036a0 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.681+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc04801a430 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.681+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06019a1e0 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.681+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0480193c0 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.681+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06019a680 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.682+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fc048019640 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.682+0000 7fc05d7fa700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.682+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 --> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc060109ec0 con 0x7fc04c038e90 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.682+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc04804d1d0 con 0x7fc060106850 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.682+0000 7fc05f7fe700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 msgr2=0x7fc04c03b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.682+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:40:19.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.883+0000 7fc05f7fe700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 msgr2=0x7fc04c03b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:13.883+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:14.284+0000 7fc05f7fe700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 msgr2=0x7fc04c03b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:14.284+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:15.085+0000 7fc05f7fe700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 msgr2=0x7fc04c03b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:15.085+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:16.686+0000 7fc05f7fe700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 msgr2=0x7fc04c03b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2152405375 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:16.686+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:17.653+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 6) v1 ==== 45695+0+0 (secure 0 0 0) 0x7fc04801a930 con 0x7fc060106850 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:17.653+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 msgr2=0x7fc04c03b350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:17.653+0000 7fc05d7fa700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.654+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 7) v1 ==== 45822+0+0 (secure 0 0 0) 0x7fc04800b9e0 con 0x7fc060106850 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.654+0000 7fc05d7fa700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fc04c03c5d0 0x7fc04c03e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.654+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc060109ec0 con 0x7fc04c03c5d0 2024-03-26T04:40:19.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.655+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fc04c03c5d0 0x7fc04c03e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.655+0000 7fc05f7fe700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fc04c03c5d0 0x7fc04c03e9c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc050003a10 tx=0x7fc050009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.656+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fc060109ec0 con 0x7fc04c03c5d0 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.662+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc06010a030 con 0x7fc04c03c5d0 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc05d7fa700 1 -- 172.21.15.12:0/3439662715 <== mgr.14120 v2:172.21.15.12:6800/284140021 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc06010a030 con 0x7fc04c03c5d0 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fc04c03c5d0 msgr2=0x7fc04c03e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fc04c03c5d0 0x7fc04c03e9c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc050003a10 tx=0x7fc050009290 comp rx=0 tx=0).stop 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 msgr2=0x7fc060199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060199aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc0480046c0 tx=0x7fc048011eb0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 shutdown_connections 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fc04c03c5d0 0x7fc04c03e9c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc060106850 0x7fc060199aa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 --2- 172.21.15.12:0/3439662715 >> [v2:172.21.15.12:6800/2152405375,v1:172.21.15.12:6801/2152405375] conn(0x7fc04c038e90 0x7fc04c03b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:19.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 >> 172.21.15.12:0/3439662715 conn(0x7fc0601002e0 msgr2=0x7fc060100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 shutdown_connections 2024-03-26T04:40:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:18.663+0000 7fc0668b7700 1 -- 172.21.15.12:0/3439662715 wait complete. 2024-03-26T04:40:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 5 is available 2024-03-26T04:40:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:Setting orchestrator backend to cephadm... 2024-03-26T04:40:19.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:19 smithi012 ceph-mon[29863]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:40:19.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:19 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:19] ENGINE Bus STARTING 2024-03-26T04:40:19.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:19 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:19] ENGINE Serving on http://172.21.15.12:8765 2024-03-26T04:40:19.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:19 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:19] ENGINE Serving on https://172.21.15.12:7150 2024-03-26T04:40:19.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:19 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:19] ENGINE Bus STARTED 2024-03-26T04:40:19.778 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:19 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe967ab700 1 Processor -- start 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe967ab700 1 -- start start 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe967ab700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90108d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe967ab700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90079630 con 0x7fbe90106930 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90108d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90108d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43966/0 (socket says 172.21.15.12:43966) 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe8ffff700 1 -- 172.21.15.12:0/938857604 learned_addr learned my addr 172.21.15.12:0/938857604 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:20.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.525+0000 7fbe8ffff700 1 -- 172.21.15.12:0/938857604 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90079770 con 0x7fbe90106930 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe8ffff700 1 --2- 172.21.15.12:0/938857604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90108d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbe800098a0 tx=0x7fbe80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcca76ff6d6e9752 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe8effd700 1 -- 172.21.15.12:0/938857604 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe80004160 con 0x7fbe90106930 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe8effd700 1 -- 172.21.15.12:0/938857604 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fbe800042c0 con 0x7fbe90106930 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe8effd700 1 -- 172.21.15.12:0/938857604 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe800118a0 con 0x7fbe90106930 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 -- 172.21.15.12:0/938857604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 msgr2=0x7fbe90108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 --2- 172.21.15.12:0/938857604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90108d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbe800098a0 tx=0x7fbe80009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 -- 172.21.15.12:0/938857604 shutdown_connections 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 --2- 172.21.15.12:0/938857604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90108d20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 -- 172.21.15.12:0/938857604 >> 172.21.15.12:0/938857604 conn(0x7fbe901002e0 msgr2=0x7fbe90102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 -- 172.21.15.12:0/938857604 shutdown_connections 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.526+0000 7fbe967ab700 1 -- 172.21.15.12:0/938857604 wait complete. 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.527+0000 7fbe967ab700 1 Processor -- start 2024-03-26T04:40:20.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.527+0000 7fbe967ab700 1 -- start start 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.527+0000 7fbe967ab700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.527+0000 7fbe967ab700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90199fe0 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.527+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43978/0 (socket says 172.21.15.12:43978) 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8ffff700 1 -- 172.21.15.12:0/2311636580 learned_addr learned my addr 172.21.15.12:0/2311636580 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8ffff700 1 -- 172.21.15.12:0/2311636580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80009550 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8ffff700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90199aa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbe800046c0 tx=0x7fbe80011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe800036a0 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9019a1e0 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fbe8001a430 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe800193c0 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.528+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9019a680 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.529+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 7) v1 ==== 45822+0+0 (secure 0 0 0) 0x7fbe80019640 con 0x7fbe90106930 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.529+0000 7fbe8d7fa700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fbe78038d80 0x7fbe7803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:20.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.529+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbe8004cd20 con 0x7fbe90106930 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.529+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe90109ed0 con 0x7fbe90106930 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.529+0000 7fbe8f7fe700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fbe78038d80 0x7fbe7803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.530+0000 7fbe8f7fe700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fbe78038d80 0x7fbe7803b240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe84006fd0 tx=0x7fbe84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.534+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbe8002d440 con 0x7fbe90106930 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.705+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fbe8001a930 con 0x7fbe90106930 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.758+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe901015b0 con 0x7fbe78038d80 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.763+0000 7fbe8d7fa700 1 -- 172.21.15.12:0/2311636580 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe901015b0 con 0x7fbe78038d80 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fbe78038d80 msgr2=0x7fbe7803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fbe78038d80 0x7fbe7803b240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe84006fd0 tx=0x7fbe84006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 msgr2=0x7fbe90199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90199aa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbe800046c0 tx=0x7fbe80011eb0 comp rx=0 tx=0).stop 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 shutdown_connections 2024-03-26T04:40:20.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fbe78038d80 0x7fbe7803b240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:20.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 --2- 172.21.15.12:0/2311636580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe90106930 0x7fbe90199aa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:20.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 >> 172.21.15.12:0/2311636580 conn(0x7fbe901002e0 msgr2=0x7fbe90100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:20.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 shutdown_connections 2024-03-26T04:40:20.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:19.766+0000 7fbe967ab700 1 -- 172.21.15.12:0/2311636580 wait complete. 2024-03-26T04:40:20.789 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:20 smithi012 ceph-mon[29863]: mgrmap e8: y(active, since 2s) 2024-03-26T04:40:20.789 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:20 smithi012 ceph-mon[29863]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:20.789 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:20 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:20.789 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:20 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.788+0000 7f167c3ab700 1 Processor -- start 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.788+0000 7f167c3ab700 1 -- start start 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.789+0000 7f167c3ab700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f1674108c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.789+0000 7f167c3ab700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1674079710 con 0x7f1674106850 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.789+0000 7f167a147700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f1674108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.789+0000 7f167a147700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f1674108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43982/0 (socket says 172.21.15.12:43982) 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.789+0000 7f167a147700 1 -- 172.21.15.12:0/1696492206 learned_addr learned my addr 172.21.15.12:0/1696492206 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.789+0000 7f167a147700 1 -- 172.21.15.12:0/1696492206 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674109180 con 0x7f1674106850 2024-03-26T04:40:21.492 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.790+0000 7f167a147700 1 --2- 172.21.15.12:0/1696492206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f1674108c40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16700098a0 tx=0x7f1670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb796725d10e671 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.790+0000 7f1679145700 1 -- 172.21.15.12:0/1696492206 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1670004160 con 0x7f1674106850 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.790+0000 7f1679145700 1 -- 172.21.15.12:0/1696492206 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f16700042c0 con 0x7f1674106850 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.790+0000 7f1679145700 1 -- 172.21.15.12:0/1696492206 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16700118a0 con 0x7f1674106850 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.790+0000 7f167c3ab700 1 -- 172.21.15.12:0/1696492206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 msgr2=0x7f1674108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.790+0000 7f167c3ab700 1 --2- 172.21.15.12:0/1696492206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f1674108c40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16700098a0 tx=0x7f1670009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 -- 172.21.15.12:0/1696492206 shutdown_connections 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 --2- 172.21.15.12:0/1696492206 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f1674108c40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 -- 172.21.15.12:0/1696492206 >> 172.21.15.12:0/1696492206 conn(0x7f16741002e0 msgr2=0x7f1674102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 -- 172.21.15.12:0/1696492206 shutdown_connections 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 -- 172.21.15.12:0/1696492206 wait complete. 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 Processor -- start 2024-03-26T04:40:21.493 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 -- start start 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f16741060d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167c3ab700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1674106610 con 0x7f1674106850 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.791+0000 7f167a147700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f16741060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f167a147700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f16741060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43988/0 (socket says 172.21.15.12:43988) 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f167a147700 1 -- 172.21.15.12:0/296279883 learned_addr learned my addr 172.21.15.12:0/296279883 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f167a147700 1 -- 172.21.15.12:0/296279883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1670009550 con 0x7f1674106850 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f167a147700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f16741060d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16700046c0 tx=0x7f1670011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16700036a0 con 0x7f1674106850 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f167001a430 con 0x7f1674106850 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1674104800 con 0x7f1674106850 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16700193c0 con 0x7f1674106850 2024-03-26T04:40:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.792+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1674104ca0 con 0x7f1674106850 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.793+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f167001a930 con 0x7f1674106850 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.793+0000 7f166b7fe700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1660038ee0 0x7f166003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.793+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1674109ec0 con 0x7f1674106850 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.793+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f167004cbd0 con 0x7f1674106850 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.793+0000 7f1679946700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1660038ee0 0x7f166003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.794+0000 7f1679946700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1660038ee0 0x7f166003b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1664006fd0 tx=0x7f1664006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:20.798+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1670019b10 con 0x7f1674106850 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.005+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f16741015c0 con 0x7f1660038ee0 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.005+0000 7f166b7fe700 1 -- 172.21.15.12:0/296279883 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f16741015c0 con 0x7f1660038ee0 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1660038ee0 msgr2=0x7f166003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1660038ee0 0x7f166003b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1664006fd0 tx=0x7f1664006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 msgr2=0x7f16741060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f16741060d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16700046c0 tx=0x7f1670011eb0 comp rx=0 tx=0).stop 2024-03-26T04:40:21.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 shutdown_connections 2024-03-26T04:40:21.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1660038ee0 0x7f166003b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:21.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 --2- 172.21.15.12:0/296279883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1674106850 0x7f16741060d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:21.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 >> 172.21.15.12:0/296279883 conn(0x7f16741002e0 msgr2=0x7f1674100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:21.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 shutdown_connections 2024-03-26T04:40:21.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.007+0000 7f167c3ab700 1 -- 172.21.15.12:0/296279883 wait complete. 2024-03-26T04:40:21.496 INFO:teuthology.orchestra.run.smithi012.stdout:Generating ssh key... 2024-03-26T04:40:21.882 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:21 smithi012 ceph-mon[29863]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: Generating public/private rsa key pair. 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: Your identification has been saved in /tmp/tmpnikmgio6/key. 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: Your public key has been saved in /tmp/tmpnikmgio6/key.pub. 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: The key fingerprint is: 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: SHA256:rZdpIQ98o9rg04sNQUuSuIvZYBUeSyiSz2vo47hkZBw ceph-c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: The key's randomart image is: 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: +---[RSA 3072]----+ 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: | ..+ | 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |+.o.+. | 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |oE.+o o | 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |. =. + o . | 2024-03-26T04:40:22.516 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |.*.. o S = | 2024-03-26T04:40:22.517 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |==o. . B = | 2024-03-26T04:40:22.517 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |++o o.o * | 2024-03-26T04:40:22.517 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |+o ..B.o | 2024-03-26T04:40:22.517 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: |+o. +.+. | 2024-03-26T04:40:22.517 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: +----[SHA256]-----+ 2024-03-26T04:40:22.798 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.881+0000 7f12a23d8700 1 Processor -- start 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f12a23d8700 1 -- start start 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f12a23d8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c104ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f12a23d8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0796b0 con 0x7f129c1046a0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f129bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c104ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f129bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c104ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44002/0 (socket says 172.21.15.12:44002) 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f129bfff700 1 -- 172.21.15.12:0/1018869230 learned_addr learned my addr 172.21.15.12:0/1018869230 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f129bfff700 1 -- 172.21.15.12:0/1018869230 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c105000 con 0x7f129c1046a0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f129bfff700 1 --2- 172.21.15.12:0/1018869230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c104ac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f12840098a0 tx=0x7f1284009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153fea8fd981d513 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.882+0000 7f129affd700 1 -- 172.21.15.12:0/1018869230 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1284004160 con 0x7f129c1046a0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f129affd700 1 -- 172.21.15.12:0/1018869230 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f12840042c0 con 0x7f129c1046a0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f129affd700 1 -- 172.21.15.12:0/1018869230 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12840118a0 con 0x7f129c1046a0 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 -- 172.21.15.12:0/1018869230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 msgr2=0x7f129c104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 --2- 172.21.15.12:0/1018869230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c104ac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f12840098a0 tx=0x7f1284009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:22.799 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 -- 172.21.15.12:0/1018869230 shutdown_connections 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 --2- 172.21.15.12:0/1018869230 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c104ac0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 -- 172.21.15.12:0/1018869230 >> 172.21.15.12:0/1018869230 conn(0x7f129c1002e0 msgr2=0x7f129c102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 -- 172.21.15.12:0/1018869230 shutdown_connections 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.883+0000 7f12a23d8700 1 -- 172.21.15.12:0/1018869230 wait complete. 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f12a23d8700 1 Processor -- start 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f12a23d8700 1 -- start start 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f12a23d8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c10a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f12a23d8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c10a680 con 0x7f129c1046a0 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f129bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c10a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f129bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c10a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44004/0 (socket says 172.21.15.12:44004) 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.884+0000 7f129bfff700 1 -- 172.21.15.12:0/1163806107 learned_addr learned my addr 172.21.15.12:0/1163806107 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f129bfff700 1 -- 172.21.15.12:0/1163806107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284009550 con 0x7f129c1046a0 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f129bfff700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c10a140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f12840046c0 tx=0x7f1284011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12840036a0 con 0x7f129c1046a0 2024-03-26T04:40:22.800 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f128401a430 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c10a880 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12840193c0 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.885+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c106c80 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.886+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f1284019640 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.886+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c107690 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.886+0000 7f12997fa700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1288038e90 0x7f128803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.886+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f128404cc50 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.886+0000 7f129b7fe700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1288038e90 0x7f128803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.886+0000 7f129b7fe700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1288038e90 0x7f128803b350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f128c006fd0 tx=0x7f128c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:21.891+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1284023e00 con 0x7f129c1046a0 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.094+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f129c106070 con 0x7f1288038e90 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.230+0000 7f12997fa700 1 -- 172.21.15.12:0/1163806107 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f129c106070 con 0x7f1288038e90 2024-03-26T04:40:22.801 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1288038e90 msgr2=0x7f128803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1288038e90 0x7f128803b350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f128c006fd0 tx=0x7f128c006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 msgr2=0x7f129c10a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c10a140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f12840046c0 tx=0x7f1284011eb0 comp rx=0 tx=0).stop 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 shutdown_connections 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f1288038e90 0x7f128803b350 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 --2- 172.21.15.12:0/1163806107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f129c1046a0 0x7f129c10a140 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 >> 172.21.15.12:0/1163806107 conn(0x7f129c1002e0 msgr2=0x7f129c192b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 shutdown_connections 2024-03-26T04:40:22.802 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:22.233+0000 7f12a23d8700 1 -- 172.21.15.12:0/1163806107 wait complete. 2024-03-26T04:40:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:23 smithi012 ceph-mon[29863]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:23 smithi012 ceph-mon[29863]: Generating ssh key... 2024-03-26T04:40:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:23 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:23 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:23.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsO79fSaetRlmZIj6pHsVEuKnWb4jxGeOHfz1uRW7cVJHQtsgmlWxzDjUSUNMieIsHH8wdUTSAOFLTPqgCMp6lCKhja1oYNR7b1i+/fKvUVycgahyz2fzUNRBtpIVPccGEZweqixj6GjmjR+6qgOaKuZlpLatKnBFRTOJlrJjC6W/ozAluhGwm4EypIjWMtQwJJNr+PlTfpg47g7GveUXrgonLNI40fT8v2SNsdb7lEqlWqVFVNKKzPhi99VuWK+N7plWUUd49DhSuTOBLoJKMfLa4nRe805LK5HETVDlvMGJ9DMWzs6R9+dvzIh0dNOVE4TPHnkym8bAiB6eN14Ci5FntsNNfIX68TD7VMDIqwbT2WArNnutRJWh2dPphAG4nt/wXMrrQPxcHGPsfgIhlz7Q8Hzaeh2FhNYFabx8dNGFwOeZASGUFoSTxDA5GEg1QDldGZmCcgckw0VQXJkgc1K0y58eZEnHCwfbaHG6mD9ioBEHMhoYnHbpPbpHfZhs= ceph-c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:40:23.914 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.203+0000 7fa521f9b700 1 Processor -- start 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.203+0000 7fa521f9b700 1 -- start start 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa521f9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c106210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa521f9b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c1067e0 con 0x7fa51c105df0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44016/0 (socket says 172.21.15.12:44016) 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51b7fe700 1 -- 172.21.15.12:0/1964710706 learned_addr learned my addr 172.21.15.12:0/1964710706 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51b7fe700 1 -- 172.21.15.12:0/1964710706 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c106ff0 con 0x7fa51c105df0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51b7fe700 1 --2- 172.21.15.12:0/1964710706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c106210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0098a0 tx=0x7fa50c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7b46cc70ddcd8a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51affd700 1 -- 172.21.15.12:0/1964710706 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c004160 con 0x7fa51c105df0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51affd700 1 -- 172.21.15.12:0/1964710706 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fa50c0042c0 con 0x7fa51c105df0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.204+0000 7fa51affd700 1 -- 172.21.15.12:0/1964710706 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c0118a0 con 0x7fa51c105df0 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 -- 172.21.15.12:0/1964710706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 msgr2=0x7fa51c106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 --2- 172.21.15.12:0/1964710706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c106210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa50c0098a0 tx=0x7fa50c009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 -- 172.21.15.12:0/1964710706 shutdown_connections 2024-03-26T04:40:23.915 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 --2- 172.21.15.12:0/1964710706 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c106210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 -- 172.21.15.12:0/1964710706 >> 172.21.15.12:0/1964710706 conn(0x7fa51c101350 msgr2=0x7fa51c1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 -- 172.21.15.12:0/1964710706 shutdown_connections 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.205+0000 7fa521f9b700 1 -- 172.21.15.12:0/1964710706 wait complete. 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa521f9b700 1 Processor -- start 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa521f9b700 1 -- start start 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa521f9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c07b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa521f9b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c07b910 con 0x7fa51c105df0 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa51b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c07b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa51b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c07b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44030/0 (socket says 172.21.15.12:44030) 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa51b7fe700 1 -- 172.21.15.12:0/3684013911 learned_addr learned my addr 172.21.15.12:0/3684013911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa51b7fe700 1 -- 172.21.15.12:0/3684013911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c009550 con 0x7fa51c105df0 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa51b7fe700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c07b3d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa50c006af0 tx=0x7fa50c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c0036a0 con 0x7fa51c105df0 2024-03-26T04:40:23.916 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fa50c01a430 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c079b00 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa50c0193c0 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.206+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c079fa0 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.207+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fa50c003800 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.207+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c0636b0 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.208+0000 7fa5197fa700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fa508038f30 0x7fa50803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.208+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa50c04ced0 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.208+0000 7fa513fff700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fa508038f30 0x7fa50803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.208+0000 7fa513fff700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fa508038f30 0x7fa50803b3f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa504006fd0 tx=0x7fa504006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.213+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa50c01a5a0 con 0x7fa51c105df0 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.418+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa51c07a3d0 con 0x7fa508038f30 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.419+0000 7fa5197fa700 1 -- 172.21.15.12:0/3684013911 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa51c07a3d0 con 0x7fa508038f30 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fa508038f30 msgr2=0x7fa50803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fa508038f30 0x7fa50803b3f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa504006fd0 tx=0x7fa504006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:23.917 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 msgr2=0x7fa51c07b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c07b3d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa50c006af0 tx=0x7fa50c0119b0 comp rx=0 tx=0).stop 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 shutdown_connections 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fa508038f30 0x7fa50803b3f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 --2- 172.21.15.12:0/3684013911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa51c105df0 0x7fa51c07b3d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 >> 172.21.15.12:0/3684013911 conn(0x7fa51c101350 msgr2=0x7fa51c101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 shutdown_connections 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:23.421+0000 7fa521f9b700 1 -- 172.21.15.12:0/3684013911 wait complete. 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T04:40:23.918 INFO:teuthology.orchestra.run.smithi012.stdout:Adding host smithi012... 2024-03-26T04:40:24.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:24 smithi012 ceph-mon[29863]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:25.484 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:25 smithi012 ceph-mon[29863]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:26.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:26 smithi012 ceph-mon[29863]: Deploying cephadm binary to smithi012 2024-03-26T04:40:29.580 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Added host 'smithi012' with addr '172.21.15.12' 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.370+0000 7f66de02d700 1 Processor -- start 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.370+0000 7f66de02d700 1 -- start start 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66de02d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d8079b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66de02d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d807a130 con 0x7f66d807b700 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66d77fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d8079b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66d77fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d8079b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44034/0 (socket says 172.21.15.12:44034) 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66d77fe700 1 -- 172.21.15.12:0/2872928250 learned_addr learned my addr 172.21.15.12:0/2872928250 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66d77fe700 1 -- 172.21.15.12:0/2872928250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d807a270 con 0x7f66d807b700 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66d77fe700 1 --2- 172.21.15.12:0/2872928250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d8079b60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb43b66ea7f10ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.371+0000 7f66d67fc700 1 -- 172.21.15.12:0/2872928250 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c8004160 con 0x7f66d807b700 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66d67fc700 1 -- 172.21.15.12:0/2872928250 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f66c80042c0 con 0x7f66d807b700 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66d67fc700 1 -- 172.21.15.12:0/2872928250 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c80118a0 con 0x7f66d807b700 2024-03-26T04:40:29.581 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 -- 172.21.15.12:0/2872928250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 msgr2=0x7f66d8079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 --2- 172.21.15.12:0/2872928250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d8079b60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f66c80098a0 tx=0x7f66c8009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 -- 172.21.15.12:0/2872928250 shutdown_connections 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 --2- 172.21.15.12:0/2872928250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d8079b60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 -- 172.21.15.12:0/2872928250 >> 172.21.15.12:0/2872928250 conn(0x7f66d81013f0 msgr2=0x7f66d8103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 -- 172.21.15.12:0/2872928250 shutdown_connections 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.372+0000 7f66de02d700 1 -- 172.21.15.12:0/2872928250 wait complete. 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66de02d700 1 Processor -- start 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66de02d700 1 -- start start 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66de02d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66de02d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d819c1f0 con 0x7f66d807b700 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66d77fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66d77fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44048/0 (socket says 172.21.15.12:44048) 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.373+0000 7f66d77fe700 1 -- 172.21.15.12:0/3809077549 learned_addr learned my addr 172.21.15.12:0/3809077549 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:29.582 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66d77fe700 1 -- 172.21.15.12:0/3809077549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c8009550 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66d77fe700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d819bcb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f66c8003f10 tx=0x7f66c8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c80036a0 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f66c801a430 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d819c3f0 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66c8024410 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.374+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d819c890 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.375+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f66c802d3f0 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.375+0000 7f66d4ff9700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f66c0038f30 0x7f66c003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.375+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f66c804cc80 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.375+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d810c100 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.375+0000 7f66d6ffd700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f66c0038f30 0x7f66c003b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.376+0000 7f66d6ffd700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f66c0038f30 0x7f66c003b3f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66cc006fd0 tx=0x7f66cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.381+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f66c8010970 con 0x7f66d807b700 2024-03-26T04:40:29.583 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:24.589+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}) v1 -- 0x7f66d8102610 con 0x7f66c0038f30 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.984+0000 7f66d4ff9700 1 -- 172.21.15.12:0/3809077549 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f66d8102610 con 0x7f66c0038f30 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f66c0038f30 msgr2=0x7f66c003b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f66c0038f30 0x7f66c003b3f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66cc006fd0 tx=0x7f66cc006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 msgr2=0x7f66d819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d819bcb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f66c8003f10 tx=0x7f66c8011940 comp rx=0 tx=0).stop 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 shutdown_connections 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f66c0038f30 0x7f66c003b3f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 --2- 172.21.15.12:0/3809077549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f66d807b700 0x7f66d819bcb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 >> 172.21.15.12:0/3809077549 conn(0x7f66d81013f0 msgr2=0x7f66d8101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 shutdown_connections 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:28.987+0000 7f66de02d700 1 -- 172.21.15.12:0/3809077549 wait complete. 2024-03-26T04:40:29.584 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mon service... 2024-03-26T04:40:30.128 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:29 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:30.128 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:29 smithi012 ceph-mon[29863]: Added host smithi012 2024-03-26T04:40:30.128 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:29 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:31.480 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:31 smithi012 ceph-mon[29863]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:31.480 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:31 smithi012 ceph-mon[29863]: Saving service mon spec with placement count:5 2024-03-26T04:40:31.480 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:31 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:31.480 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:31 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.062+0000 7f697e0d0700 1 Processor -- start 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.062+0000 7f697e0d0700 1 -- start start 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f697e0d0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f6978105120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f697e0d0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69781056f0 con 0x7f6978104d00 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f6978105120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f6978105120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46368/0 (socket says 172.21.15.12:46368) 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69777fe700 1 -- 172.21.15.12:0/3708711981 learned_addr learned my addr 172.21.15.12:0/3708711981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69777fe700 1 -- 172.21.15.12:0/3708711981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6978105f00 con 0x7f6978104d00 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69777fe700 1 --2- 172.21.15.12:0/3708711981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f6978105120 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f69680098a0 tx=0x7f6968009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855095f6c60ad10b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:31.755 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69767fc700 1 -- 172.21.15.12:0/3708711981 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6968004160 con 0x7f6978104d00 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69767fc700 1 -- 172.21.15.12:0/3708711981 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f69680042c0 con 0x7f6978104d00 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.063+0000 7f69767fc700 1 -- 172.21.15.12:0/3708711981 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69680118a0 con 0x7f6978104d00 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 -- 172.21.15.12:0/3708711981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 msgr2=0x7f6978105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 --2- 172.21.15.12:0/3708711981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f6978105120 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f69680098a0 tx=0x7f6968009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 -- 172.21.15.12:0/3708711981 shutdown_connections 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 --2- 172.21.15.12:0/3708711981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f6978105120 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 -- 172.21.15.12:0/3708711981 >> 172.21.15.12:0/3708711981 conn(0x7f69781002a0 msgr2=0x7f69781026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 -- 172.21.15.12:0/3708711981 shutdown_connections 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.064+0000 7f697e0d0700 1 -- 172.21.15.12:0/3708711981 wait complete. 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f697e0d0700 1 Processor -- start 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f697e0d0700 1 -- start start 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f697e0d0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f69781989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f697e0d0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6978198ee0 con 0x7f6978104d00 2024-03-26T04:40:31.756 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f69777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f69781989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f69777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f69781989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46380/0 (socket says 172.21.15.12:46380) 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f69777fe700 1 -- 172.21.15.12:0/2287647733 learned_addr learned my addr 172.21.15.12:0/2287647733 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f69777fe700 1 -- 172.21.15.12:0/2287647733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009550 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.065+0000 7f69777fe700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f69781989a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6968003f10 tx=0x7f6968011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.066+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69680036a0 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.066+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f696801a430 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.066+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69781990e0 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.066+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6968024410 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.066+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6978199580 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.066+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f696802d3f0 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.067+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69780636b0 con 0x7f6978104d00 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.067+0000 7f6974ff9700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6964038ee0 0x7f696403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:31.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.067+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f696804ccb0 con 0x7f6978104d00 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.067+0000 7f6976ffd700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6964038ee0 0x7f696403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.067+0000 7f6976ffd700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6964038ee0 0x7f696403b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6960006fd0 tx=0x7f6960006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.072+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f696801a5a0 con 0x7f6978104d00 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.293+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6978101510 con 0x7f6964038ee0 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.297+0000 7f6974ff9700 1 -- 172.21.15.12:0/2287647733 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6978101510 con 0x7f6964038ee0 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6964038ee0 msgr2=0x7f696403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6964038ee0 0x7f696403b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6960006fd0 tx=0x7f6960006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 msgr2=0x7f69781989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f69781989a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6968003f10 tx=0x7f6968011940 comp rx=0 tx=0).stop 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 shutdown_connections 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6964038ee0 0x7f696403b3a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 --2- 172.21.15.12:0/2287647733 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6978104d00 0x7f69781989a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:31.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.299+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 >> 172.21.15.12:0/2287647733 conn(0x7f69781002a0 msgr2=0x7f6978100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:31.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.300+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 shutdown_connections 2024-03-26T04:40:31.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:30.300+0000 7f697e0d0700 1 -- 172.21.15.12:0/2287647733 wait complete. 2024-03-26T04:40:31.759 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mgr service... 2024-03-26T04:40:33.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:32 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:33.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:32 smithi012 ceph-mon[29863]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:33.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:32 smithi012 ceph-mon[29863]: Saving service mgr spec with placement count:2 2024-03-26T04:40:33.114 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:32 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2c17eb700 1 Processor -- start 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2c17eb700 1 -- start start 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2c17eb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc108c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2c17eb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc0796d0 con 0x7fb2bc106880 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2baffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc108c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2baffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc108c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46396/0 (socket says 172.21.15.12:46396) 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2baffd700 1 -- 172.21.15.12:0/606661612 learned_addr learned my addr 172.21.15.12:0/606661612 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.219+0000 7fb2baffd700 1 -- 172.21.15.12:0/606661612 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc1091b0 con 0x7fb2bc106880 2024-03-26T04:40:33.607 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2baffd700 1 --2- 172.21.15.12:0/606661612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc108c70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0098a0 tx=0x7fb2ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deee61a36ce258ec server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2b9ffb700 1 -- 172.21.15.12:0/606661612 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac004160 con 0x7fb2bc106880 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2b9ffb700 1 -- 172.21.15.12:0/606661612 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb2ac0042c0 con 0x7fb2bc106880 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2b9ffb700 1 -- 172.21.15.12:0/606661612 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac011850 con 0x7fb2bc106880 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2c17eb700 1 -- 172.21.15.12:0/606661612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 msgr2=0x7fb2bc108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2c17eb700 1 --2- 172.21.15.12:0/606661612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc108c70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0098a0 tx=0x7fb2ac009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2c17eb700 1 -- 172.21.15.12:0/606661612 shutdown_connections 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2c17eb700 1 --2- 172.21.15.12:0/606661612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc108c70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2c17eb700 1 -- 172.21.15.12:0/606661612 >> 172.21.15.12:0/606661612 conn(0x7fb2bc100290 msgr2=0x7fb2bc1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.220+0000 7fb2c17eb700 1 -- 172.21.15.12:0/606661612 shutdown_connections 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.221+0000 7fb2c17eb700 1 -- 172.21.15.12:0/606661612 wait complete. 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.221+0000 7fb2c17eb700 1 Processor -- start 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.221+0000 7fb2c17eb700 1 -- start start 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.221+0000 7fb2c17eb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.221+0000 7fb2c17eb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc199f80 con 0x7fb2bc106880 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2baffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:33.608 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2baffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46402/0 (socket says 172.21.15.12:46402) 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2baffd700 1 -- 172.21.15.12:0/2716537405 learned_addr learned my addr 172.21.15.12:0/2716537405 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2baffd700 1 -- 172.21.15.12:0/2716537405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac009550 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2baffd700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc199a40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac006af0 tx=0x7fb2ac011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac0036a0 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc19a180 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb2ac01a430 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2ac0193c0 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.222+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc19a620 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.223+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7fb2ac019680 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.223+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc109ed0 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.223+0000 7fb2a3fff700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fb2a4038e90 0x7fb2a403b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.223+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb2ac04cce0 con 0x7fb2bc106880 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.223+0000 7fb2ba7fc700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fb2a4038e90 0x7fb2a403b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:33.609 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.224+0000 7fb2ba7fc700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fb2a4038e90 0x7fb2a403b350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0006fd0 tx=0x7fb2b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.228+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb2ac018350 con 0x7fb2bc106880 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.466+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb2bc1014b0 con 0x7fb2a4038e90 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.470+0000 7fb2a3fff700 1 -- 172.21.15.12:0/2716537405 <== mgr.14120 v2:172.21.15.12:6800/284140021 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb2bc1014b0 con 0x7fb2a4038e90 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fb2a4038e90 msgr2=0x7fb2a403b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fb2a4038e90 0x7fb2a403b350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0006fd0 tx=0x7fb2b0006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 msgr2=0x7fb2bc199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc199a40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac006af0 tx=0x7fb2ac011a50 comp rx=0 tx=0).stop 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 shutdown_connections 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fb2a4038e90 0x7fb2a403b350 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 --2- 172.21.15.12:0/2716537405 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb2bc106880 0x7fb2bc199a40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.472+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 >> 172.21.15.12:0/2716537405 conn(0x7fb2bc100290 msgr2=0x7fb2bc100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.473+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 shutdown_connections 2024-03-26T04:40:33.610 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:32.473+0000 7fb2c17eb700 1 -- 172.21.15.12:0/2716537405 wait complete. 2024-03-26T04:40:34.256 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:34 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:34.830 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.040+0000 7f628c5da700 1 Processor -- start 2024-03-26T04:40:34.830 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.040+0000 7f628c5da700 1 -- start start 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.040+0000 7f628c5da700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f62841044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.040+0000 7f628c5da700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6284104ac0 con 0x7f62841040d0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f628a376700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f62841044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f628a376700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f62841044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46408/0 (socket says 172.21.15.12:46408) 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f628a376700 1 -- 172.21.15.12:0/3901103814 learned_addr learned my addr 172.21.15.12:0/3901103814 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f628a376700 1 -- 172.21.15.12:0/3901103814 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284107060 con 0x7f62841040d0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f628a376700 1 --2- 172.21.15.12:0/3901103814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f62841044f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f62740098a0 tx=0x7f6274009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb210f26827bf02 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f6289374700 1 -- 172.21.15.12:0/3901103814 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6274004160 con 0x7f62841040d0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f6289374700 1 -- 172.21.15.12:0/3901103814 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f62740042c0 con 0x7f62841040d0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.041+0000 7f6289374700 1 -- 172.21.15.12:0/3901103814 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6274011850 con 0x7f62841040d0 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 -- 172.21.15.12:0/3901103814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 msgr2=0x7f62841044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 --2- 172.21.15.12:0/3901103814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f62841044f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f62740098a0 tx=0x7f6274009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:34.831 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 -- 172.21.15.12:0/3901103814 shutdown_connections 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 --2- 172.21.15.12:0/3901103814 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f62841044f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 -- 172.21.15.12:0/3901103814 >> 172.21.15.12:0/3901103814 conn(0x7f62840ff9e0 msgr2=0x7f6284101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 -- 172.21.15.12:0/3901103814 shutdown_connections 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.042+0000 7f628c5da700 1 -- 172.21.15.12:0/3901103814 wait complete. 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628c5da700 1 Processor -- start 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628c5da700 1 -- start start 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628c5da700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f6284199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628c5da700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628419a000 con 0x7f62841040d0 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628a376700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f6284199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628a376700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f6284199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46416/0 (socket says 172.21.15.12:46416) 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628a376700 1 -- 172.21.15.12:0/2157159615 learned_addr learned my addr 172.21.15.12:0/2157159615 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628a376700 1 -- 172.21.15.12:0/2157159615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274009550 con 0x7f62841040d0 2024-03-26T04:40:34.832 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.043+0000 7f628a376700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f6284199ac0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6274003d50 tx=0x7f6274011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.044+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62740037d0 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.044+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6274003930 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.044+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62740193c0 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.044+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628419a200 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.044+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628419a6a0 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.045+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f627401a430 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.045+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62840636b0 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.045+0000 7f627b7fe700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6270038ee0 0x7f627003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.045+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f627404cf90 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.045+0000 7f6289b75700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6270038ee0 0x7f627003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.045+0000 7f6289b75700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6270038ee0 0x7f627003b3a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6280006fd0 tx=0x7f6280006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.050+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6274081280 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.255+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f628419cf80 con 0x7f62841040d0 2024-03-26T04:40:34.833 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.259+0000 7f627b7fe700 1 -- 172.21.15.12:0/2157159615 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6274003aa0 con 0x7f62841040d0 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6270038ee0 msgr2=0x7f627003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6270038ee0 0x7f627003b3a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6280006fd0 tx=0x7f6280006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 msgr2=0x7f6284199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f6284199ac0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6274003d50 tx=0x7f6274011ac0 comp rx=0 tx=0).stop 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 shutdown_connections 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f6270038ee0 0x7f627003b3a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 --2- 172.21.15.12:0/2157159615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62841040d0 0x7f6284199ac0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 >> 172.21.15.12:0/2157159615 conn(0x7f62840ff9e0 msgr2=0x7f6284100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 shutdown_connections 2024-03-26T04:40:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:34.261+0000 7f628c5da700 1 -- 172.21.15.12:0/2157159615 wait complete. 2024-03-26T04:40:35.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2157159615' entity='client.admin' 2024-03-26T04:40:35.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:35 smithi012 ceph-mon[29863]: from='mgr.14120 172.21.15.12:0/2816555128' entity='mgr.y' 2024-03-26T04:40:36.585 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/760242035' entity='client.admin' 2024-03-26T04:40:36.812 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.215+0000 7f43329a6700 1 Processor -- start 2024-03-26T04:40:36.812 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.215+0000 7f43329a6700 1 -- start start 2024-03-26T04:40:36.812 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.215+0000 7f43329a6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c108c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:36.812 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.215+0000 7f43329a6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c078e80 con 0x7f432c106850 2024-03-26T04:40:36.812 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.216+0000 7f432bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.216+0000 7f432bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46420/0 (socket says 172.21.15.12:46420) 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.216+0000 7f432bfff700 1 -- 172.21.15.12:0/2189311282 learned_addr learned my addr 172.21.15.12:0/2189311282 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.216+0000 7f432bfff700 1 -- 172.21.15.12:0/2189311282 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c109180 con 0x7f432c106850 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.216+0000 7f432bfff700 1 --2- 172.21.15.12:0/2189311282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c108c40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f431c0098a0 tx=0x7f431c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7838eeabe52c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.217+0000 7f432affd700 1 -- 172.21.15.12:0/2189311282 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c004160 con 0x7f432c106850 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.217+0000 7f432affd700 1 -- 172.21.15.12:0/2189311282 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f431c0042c0 con 0x7f432c106850 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.217+0000 7f432affd700 1 -- 172.21.15.12:0/2189311282 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c011850 con 0x7f432c106850 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 -- 172.21.15.12:0/2189311282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 msgr2=0x7f432c108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 --2- 172.21.15.12:0/2189311282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c108c40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f431c0098a0 tx=0x7f431c009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 -- 172.21.15.12:0/2189311282 shutdown_connections 2024-03-26T04:40:36.813 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 --2- 172.21.15.12:0/2189311282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c108c40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 -- 172.21.15.12:0/2189311282 >> 172.21.15.12:0/2189311282 conn(0x7f432c100260 msgr2=0x7f432c1026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 -- 172.21.15.12:0/2189311282 shutdown_connections 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 -- 172.21.15.12:0/2189311282 wait complete. 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 Processor -- start 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.218+0000 7f43329a6700 1 -- start start 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f43329a6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c1060d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f43329a6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c106610 con 0x7f432c106850 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f432bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c1060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f432bfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c1060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46432/0 (socket says 172.21.15.12:46432) 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f432bfff700 1 -- 172.21.15.12:0/760242035 learned_addr learned my addr 172.21.15.12:0/760242035 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:36.814 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f432bfff700 1 -- 172.21.15.12:0/760242035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c009550 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f432bfff700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c1060d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f431c003ec0 tx=0x7f431c011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.219+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c0037d0 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.220+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f431c003930 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.220+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c104800 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.220+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c104ca0 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.220+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f431c0193c0 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.220+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f431c019650 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.221+0000 7f43297fa700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f4314038ee0 0x7f431403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.221+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f431c04de30 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.221+0000 7f432b7fe700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f4314038ee0 0x7f431403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.221+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4318005150 con 0x7f432c106850 2024-03-26T04:40:36.815 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.221+0000 7f432b7fe700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f4314038ee0 0x7f431403b3a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4320006fd0 tx=0x7f4320006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.226+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f431c048020 con 0x7f432c106850 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.437+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f4318005980 con 0x7f432c106850 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.441+0000 7f43297fa700 1 -- 172.21.15.12:0/760242035 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f431c00b3e0 con 0x7f432c106850 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f4314038ee0 msgr2=0x7f431403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f4314038ee0 0x7f431403b3a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4320006fd0 tx=0x7f4320006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 msgr2=0x7f432c1060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c1060d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f431c003ec0 tx=0x7f431c011980 comp rx=0 tx=0).stop 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 shutdown_connections 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f4314038ee0 0x7f431403b3a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 --2- 172.21.15.12:0/760242035 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f432c106850 0x7f432c1060d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 >> 172.21.15.12:0/760242035 conn(0x7f432c100260 msgr2=0x7f432c100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:36.816 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 shutdown_connections 2024-03-26T04:40:36.817 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:35.444+0000 7f43329a6700 1 -- 172.21.15.12:0/760242035 wait complete. 2024-03-26T04:40:36.817 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling the dashboard module... 2024-03-26T04:40:37.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2873642827' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T04:40:37.905 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:37 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ignoring --setuser ceph since I am not root 2024-03-26T04:40:37.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:37 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ignoring --setgroup ceph since I am not root 2024-03-26T04:40:38.226 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:38 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:38.130+0000 7f05d1396200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:40:38.234 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.227+0000 7f2ba64eb700 1 Processor -- start 2024-03-26T04:40:38.234 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.227+0000 7f2ba64eb700 1 -- start start 2024-03-26T04:40:38.234 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.227+0000 7f2ba64eb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba0108c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:38.234 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.227+0000 7f2ba64eb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba0079710 con 0x7f2ba0106850 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.227+0000 7f2b9ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba0108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba0108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46434/0 (socket says 172.21.15.12:46434) 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9ffff700 1 -- 172.21.15.12:0/2746403037 learned_addr learned my addr 172.21.15.12:0/2746403037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9ffff700 1 -- 172.21.15.12:0/2746403037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba0109180 con 0x7f2ba0106850 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9ffff700 1 --2- 172.21.15.12:0/2746403037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba0108c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b900098a0 tx=0x7f2b90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0bddfc2407205b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9effd700 1 -- 172.21.15.12:0/2746403037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b90004160 con 0x7f2ba0106850 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9effd700 1 -- 172.21.15.12:0/2746403037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2b900042c0 con 0x7f2ba0106850 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.228+0000 7f2b9effd700 1 -- 172.21.15.12:0/2746403037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b90011850 con 0x7f2ba0106850 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2746403037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 msgr2=0x7f2ba0108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 --2- 172.21.15.12:0/2746403037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba0108c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2b900098a0 tx=0x7f2b90009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2746403037 shutdown_connections 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 --2- 172.21.15.12:0/2746403037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba0108c40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:38.235 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2746403037 >> 172.21.15.12:0/2746403037 conn(0x7f2ba01002e0 msgr2=0x7f2ba0102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2746403037 shutdown_connections 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2746403037 wait complete. 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 Processor -- start 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.229+0000 7f2ba64eb700 1 -- start start 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2ba64eb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba01060d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2ba64eb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba0106610 con 0x7f2ba0106850 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2b9ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba01060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2b9ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba01060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46446/0 (socket says 172.21.15.12:46446) 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2b9ffff700 1 -- 172.21.15.12:0/2873642827 learned_addr learned my addr 172.21.15.12:0/2873642827 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2b9ffff700 1 -- 172.21.15.12:0/2873642827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90009550 con 0x7f2ba0106850 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.230+0000 7f2b9ffff700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba01060d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006af0 tx=0x7f2b90011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.231+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b900036a0 con 0x7f2ba0106850 2024-03-26T04:40:38.236 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.231+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2b9001a430 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.231+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba0104800 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.231+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b9001ae60 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.231+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba0104ca0 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.231+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) v1 ==== 45928+0+0 (secure 0 0 0) 0x7f2b90003800 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.232+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba0109f00 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.232+0000 7f2b9d7fa700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f2b88038e90 0x7f2b8803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.232+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2b9004cf00 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.232+0000 7f2b9f7fe700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f2b88038e90 0x7f2b8803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.232+0000 7f2b9f7fe700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f2b88038e90 0x7f2b8803b350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b94006fd0 tx=0x7f2b94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.237+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2b90018350 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.500+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f2ba010a110 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.552+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f2b90023c20 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.553+0000 7f2b9d7fa700 1 -- 172.21.15.12:0/2873642827 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 9) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f2b90019820 con 0x7f2ba0106850 2024-03-26T04:40:38.237 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f2b88038e90 msgr2=0x7f2b8803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f2b88038e90 0x7f2b8803b350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b94006fd0 tx=0x7f2b94006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 msgr2=0x7f2ba01060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba01060d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b90006af0 tx=0x7f2b90011a30 comp rx=0 tx=0).stop 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 shutdown_connections 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f2b88038e90 0x7f2b8803b350 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 --2- 172.21.15.12:0/2873642827 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ba0106850 0x7f2ba01060d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 >> 172.21.15.12:0/2873642827 conn(0x7f2ba01002e0 msgr2=0x7f2ba0100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 shutdown_connections 2024-03-26T04:40:38.238 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:37.555+0000 7f2ba64eb700 1 -- 172.21.15.12:0/2873642827 wait complete. 2024-03-26T04:40:38.529 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:38 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:38.227+0000 7f05d1396200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:40:38.835 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2873642827' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T04:40:38.835 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:38 smithi012 ceph-mon[29863]: mgrmap e9: y(active, since 19s) 2024-03-26T04:40:39.142 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:38 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:38.897+0000 7f05d1396200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:40:39.142 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:38 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:38.996+0000 7f05d1396200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:40:39.142 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.141+0000 7f05d1396200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:40:39.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.240+0000 7f05d1396200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:40:39.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.326+0000 7f05d1396200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.629+0000 7fcdae6df700 1 Processor -- start 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.629+0000 7fcdae6df700 1 -- start start 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.629+0000 7fcdae6df700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8108c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:39.533 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.629+0000 7fcdae6df700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80796d0 con 0x7fcda8106890 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8108c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8108c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46482/0 (socket says 172.21.15.12:46482) 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda7fff700 1 -- 172.21.15.12:0/3277425442 learned_addr learned my addr 172.21.15.12:0/3277425442 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda7fff700 1 -- 172.21.15.12:0/3277425442 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda81091c0 con 0x7fcda8106890 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda7fff700 1 --2- 172.21.15.12:0/3277425442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8108c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098a0 tx=0x7fcd9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68833a12dca7f44b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda6ffd700 1 -- 172.21.15.12:0/3277425442 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c004160 con 0x7fcda8106890 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda6ffd700 1 -- 172.21.15.12:0/3277425442 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcd9c0042c0 con 0x7fcda8106890 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.630+0000 7fcda6ffd700 1 -- 172.21.15.12:0/3277425442 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c011850 con 0x7fcda8106890 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 -- 172.21.15.12:0/3277425442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 msgr2=0x7fcda8108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 --2- 172.21.15.12:0/3277425442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8108c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098a0 tx=0x7fcd9c009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 -- 172.21.15.12:0/3277425442 shutdown_connections 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 --2- 172.21.15.12:0/3277425442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8108c80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 -- 172.21.15.12:0/3277425442 >> 172.21.15.12:0/3277425442 conn(0x7fcda81002e0 msgr2=0x7fcda8102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:39.534 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 -- 172.21.15.12:0/3277425442 shutdown_connections 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.631+0000 7fcdae6df700 1 -- 172.21.15.12:0/3277425442 wait complete. 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcdae6df700 1 Processor -- start 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcdae6df700 1 -- start start 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcdae6df700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcdae6df700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda819a010 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46492/0 (socket says 172.21.15.12:46492) 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcda7fff700 1 -- 172.21.15.12:0/1634129071 learned_addr learned my addr 172.21.15.12:0/1634129071 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.632+0000 7fcda7fff700 1 -- 172.21.15.12:0/1634129071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c009550 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.633+0000 7fcda7fff700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8199ad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c006af0 tx=0x7fcd9c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.633+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c0036a0 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.633+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcd9c01a430 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.633+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda819a210 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.633+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c01ae60 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.633+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda819a6b0 con 0x7fcda8106890 2024-03-26T04:40:39.535 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.634+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 45941+0+0 (secure 0 0 0) 0x7fcd9c0194a0 con 0x7fcda8106890 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.634+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda8109ee0 con 0x7fcda8106890 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.634+0000 7fcda57fa700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 0x7fcd9003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.634+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcd9c04cfb0 con 0x7fcda8106890 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.634+0000 7fcda77fe700 1 -- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 msgr2=0x7fcd9003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.634+0000 7fcda77fe700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 0x7fcd9003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.639+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcd9c081280 con 0x7fcda8106890 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.835+0000 7fcda77fe700 1 -- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 msgr2=0x7fcd9003b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.835+0000 7fcda77fe700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 0x7fcd9003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.902+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcda819d090 con 0x7fcda8106890 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.902+0000 7fcda57fa700 1 -- 172.21.15.12:0/1634129071 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fcd9c02b340 con 0x7fcda8106890 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 msgr2=0x7fcd9003b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 0x7fcd9003b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 msgr2=0x7fcda8199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:39.536 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8199ad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c006af0 tx=0x7fcd9c011a30 comp rx=0 tx=0).stop 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 shutdown_connections 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7fcd90038ee0 0x7fcd9003b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 --2- 172.21.15.12:0/1634129071 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcda8106890 0x7fcda8199ad0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 >> 172.21.15.12:0/1634129071 conn(0x7fcda81002e0 msgr2=0x7fcda8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 shutdown_connections 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:38.905+0000 7fcdae6df700 1 -- 172.21.15.12:0/1634129071 wait complete. 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-03-26T04:40:39.537 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 9... 2024-03-26T04:40:39.796 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1634129071' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:40:39.796 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.600+0000 7f05d1396200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:40:39.796 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.716+0000 7f05d1396200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:40.088 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.808+0000 7f05d1396200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:40:40.088 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:39 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:39.908+0000 7f05d1396200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:40:40.088 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:40 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:40.087+0000 7f05d1396200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:40:41.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:41 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:41.070+0000 7f05d1396200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:40:41.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:41 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:41.204+0000 7f05d1396200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:40:41.836 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:41 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:41.543+0000 7f05d1396200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:40:41.836 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:41 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:41.634+0000 7f05d1396200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:40:41.836 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:41 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:41.835+0000 7f05d1396200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:42.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:42.026+0000 7f05d1396200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:40:42.427 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:42.320+0000 7f05d1396200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:42.840 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:42.590+0000 7f05d1396200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:42.840 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:42.840+0000 7f05d1396200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:40:44.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:43.670+0000 7f05d1396200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:40:44.573 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:44.250+0000 7f05d1396200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:40:44.573 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:44.350+0000 7f05d1396200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:40:44.573 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:40:44.573+0000 7f05d1396200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: Active manager daemon y restarted 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: Activating manager daemon y 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: mgrmap e10: y(active, starting, since 0.00322248s) 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:40:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:40:44.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:40:44.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:40:44.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: Manager daemon y is now available 2024-03-26T04:40:44.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:44.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:44] ENGINE Bus STARTING 2024-03-26T04:40:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:44] ENGINE Serving on https://172.21.15.12:7150 2024-03-26T04:40:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:45] ENGINE Serving on http://172.21.15.12:8765 2024-03-26T04:40:45.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: [26/Mar/2024:04:40:45] ENGINE Bus STARTED 2024-03-26T04:40:45.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:40:45.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: mgrmap e11: y(active, since 1.00567s) 2024-03-26T04:40:45.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:45 smithi012 ceph-mon[29863]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:40:46.495 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.896+0000 7f7165f5c700 1 Processor -- start 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.896+0000 7f7165f5c700 1 -- start start 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.896+0000 7f7165f5c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160105100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.896+0000 7f7165f5c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71601056d0 con 0x7f7160104ce0 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715f7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160105100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715f7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160105100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42284/0 (socket says 172.21.15.12:42284) 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715f7fe700 1 -- 172.21.15.12:0/1805680760 learned_addr learned my addr 172.21.15.12:0/1805680760 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715f7fe700 1 -- 172.21.15.12:0/1805680760 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7160105ee0 con 0x7f7160104ce0 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715f7fe700 1 --2- 172.21.15.12:0/1805680760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160105100 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7150009ab0 tx=0x7f7150009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fb362ad558be39 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:46.496 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715effd700 1 -- 172.21.15.12:0/1805680760 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7150004160 con 0x7f7160104ce0 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715effd700 1 -- 172.21.15.12:0/1805680760 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f715000bc60 con 0x7f7160104ce0 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.897+0000 7f715effd700 1 -- 172.21.15.12:0/1805680760 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7150003a70 con 0x7f7160104ce0 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 -- 172.21.15.12:0/1805680760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 msgr2=0x7f7160105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1805680760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160105100 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7150009ab0 tx=0x7f7150009dc0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 -- 172.21.15.12:0/1805680760 shutdown_connections 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1805680760 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160105100 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 -- 172.21.15.12:0/1805680760 >> 172.21.15.12:0/1805680760 conn(0x7f71601002c0 msgr2=0x7f71601026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 -- 172.21.15.12:0/1805680760 shutdown_connections 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.898+0000 7f7165f5c700 1 -- 172.21.15.12:0/1805680760 wait complete. 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f7165f5c700 1 Processor -- start 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f7165f5c700 1 -- start start 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f7165f5c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160079310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:46.497 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f7165f5c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716019c7a0 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f715f7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160079310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f715f7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160079310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42296/0 (socket says 172.21.15.12:42296) 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f715f7fe700 1 -- 172.21.15.12:0/1773464547 learned_addr learned my addr 172.21.15.12:0/1773464547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.899+0000 7f715f7fe700 1 -- 172.21.15.12:0/1773464547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150009550 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.900+0000 7f715f7fe700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160079310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f71500037e0 tx=0x7f7150003bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.900+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f715000bdd0 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.900+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7150004350 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.900+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716019c8e0 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.900+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7150011420 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.900+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716019ccc0 con 0x7f7160104ce0 2024-03-26T04:40:46.498 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.901+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f715001a430 con 0x7f7160104ce0 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.901+0000 7f715d7fa700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.901+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 --> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f714c03bb60 con 0x7f714c038f90 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.901+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f715004d070 con 0x7f7160104ce0 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.901+0000 7f7156ffd700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 msgr2=0x7f714c03b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:39.901+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:40.102+0000 7f7156ffd700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 msgr2=0x7f714c03b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:40.102+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:40.503+0000 7f7156ffd700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 msgr2=0x7f714c03b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:40.503+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:41.304+0000 7f7156ffd700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 msgr2=0x7f714c03b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:41.304+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:42.905+0000 7f7156ffd700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 msgr2=0x7f714c03b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/284140021 2024-03-26T04:40:46.499 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:42.905+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:44.588+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 10) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f7150011580 con 0x7f7160104ce0 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:44.588+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 msgr2=0x7f714c03b450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:44.588+0000 7f715d7fa700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.589+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 11) v1 ==== 45835+0+0 (secure 0 0 0) 0x7f715004ed40 con 0x7f7160104ce0 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.589+0000 7f715d7fa700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f714c03ca40 0x7f714c03ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.590+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f714c03bb60 con 0x7f714c03ca40 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.590+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f714c03ca40 0x7f714c03ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.591+0000 7f7156ffd700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f714c03ca40 0x7f714c03ee30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7148003a10 tx=0x7f7148009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.591+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f714c03bb60 con 0x7f714c03ca40 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.598+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f71600636b0 con 0x7f714c03ca40 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.598+0000 7f715d7fa700 1 -- 172.21.15.12:0/1773464547 <== mgr.14152 v2:172.21.15.12:6800/1454619140 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f71600636b0 con 0x7f714c03ca40 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.598+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f714c03ca40 msgr2=0x7f714c03ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.598+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f714c03ca40 0x7f714c03ee30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7148003a10 tx=0x7f7148009270 comp rx=0 tx=0).stop 2024-03-26T04:40:46.500 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.598+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 msgr2=0x7f7160079310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.598+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160079310 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f71500037e0 tx=0x7f7150003bc0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 shutdown_connections 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f714c03ca40 0x7f714c03ee30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7160104ce0 0x7f7160079310 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 --2- 172.21.15.12:0/1773464547 >> [v2:172.21.15.12:6800/284140021,v1:172.21.15.12:6801/284140021] conn(0x7f714c038f90 0x7f714c03b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 >> 172.21.15.12:0/1773464547 conn(0x7f71601002c0 msgr2=0x7f7160100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 shutdown_connections 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:45.599+0000 7f7165f5c700 1 -- 172.21.15.12:0/1773464547 wait complete. 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 9 is available 2024-03-26T04:40:46.501 INFO:teuthology.orchestra.run.smithi012.stdout:Generating a dashboard self-signed certificate... 2024-03-26T04:40:47.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.917+0000 7f1500d45700 1 Processor -- start 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.917+0000 7f1500d45700 1 -- start start 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f1500d45700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc105100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f1500d45700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc1056d0 con 0x7f14fc104ce0 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14fa59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc105100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:47.975 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14fa59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc105100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42360/0 (socket says 172.21.15.12:42360) 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14fa59c700 1 -- 172.21.15.12:0/3922390111 learned_addr learned my addr 172.21.15.12:0/3922390111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14fa59c700 1 -- 172.21.15.12:0/3922390111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc105ee0 con 0x7f14fc104ce0 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14fa59c700 1 --2- 172.21.15.12:0/3922390111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc105100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00bb40 tx=0x7f14ec00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf460791c02109a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14f959a700 1 -- 172.21.15.12:0/3922390111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ec00dbe0 con 0x7f14fc104ce0 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14f959a700 1 -- 172.21.15.12:0/3922390111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f14ec0044b0 con 0x7f14fc104ce0 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.918+0000 7f14f959a700 1 -- 172.21.15.12:0/3922390111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ec003cd0 con 0x7f14fc104ce0 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 -- 172.21.15.12:0/3922390111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 msgr2=0x7f14fc105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 --2- 172.21.15.12:0/3922390111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc105100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00bb40 tx=0x7f14ec00be50 comp rx=0 tx=0).stop 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 -- 172.21.15.12:0/3922390111 shutdown_connections 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 --2- 172.21.15.12:0/3922390111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc105100 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 -- 172.21.15.12:0/3922390111 >> 172.21.15.12:0/3922390111 conn(0x7f14fc100260 msgr2=0x7f14fc1026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 -- 172.21.15.12:0/3922390111 shutdown_connections 2024-03-26T04:40:47.976 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.919+0000 7f1500d45700 1 -- 172.21.15.12:0/3922390111 wait complete. 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f1500d45700 1 Processor -- start 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f1500d45700 1 -- start start 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f1500d45700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f1500d45700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc197d70 con 0x7f14fc104ce0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f14fa59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f14fa59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42376/0 (socket says 172.21.15.12:42376) 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.920+0000 7f14fa59c700 1 -- 172.21.15.12:0/2711600357 learned_addr learned my addr 172.21.15.12:0/2711600357 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f14fa59c700 1 -- 172.21.15.12:0/2711600357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec00b7f0 con 0x7f14fc104ce0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f14fa59c700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc197830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f14ec003a40 tx=0x7f14ec00dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ec01e070 con 0x7f14fc104ce0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f14ec010b00 con 0x7f14fc104ce0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc197f70 con 0x7f14fc104ce0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ec022970 con 0x7f14fc104ce0 2024-03-26T04:40:47.977 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.921+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc198410 con 0x7f14fc104ce0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.922+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 11) v1 ==== 45835+0+0 (secure 0 0 0) 0x7f14ec02c440 con 0x7f14fc104ce0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.922+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc0636b0 con 0x7f14fc104ce0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.922+0000 7f14eb7fe700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f14e4038e10 0x7f14e403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.923+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f14ec04f9d0 con 0x7f14fc104ce0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.923+0000 7f14f9d9b700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f14e4038e10 0x7f14e403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.923+0000 7f14f9d9b700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f14e4038e10 0x7f14e403b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14f0006fd0 tx=0x7f14f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:46.928+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f14ec02a950 con 0x7f14fc104ce0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.137+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f14fc0624c0 con 0x7f14e4038e10 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.204+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 12) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f14ec0193f0 con 0x7f14fc104ce0 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.353+0000 7f14eb7fe700 1 -- 172.21.15.12:0/2711600357 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f14fc0624c0 con 0x7f14e4038e10 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f14e4038e10 msgr2=0x7f14e403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f14e4038e10 0x7f14e403b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14f0006fd0 tx=0x7f14f0006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 msgr2=0x7f14fc197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:47.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc197830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f14ec003a40 tx=0x7f14ec00dfd0 comp rx=0 tx=0).stop 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 shutdown_connections 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f14e4038e10 0x7f14e403b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 --2- 172.21.15.12:0/2711600357 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f14fc104ce0 0x7f14fc197830 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 >> 172.21.15.12:0/2711600357 conn(0x7f14fc100260 msgr2=0x7f14fc18e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 shutdown_connections 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:47.355+0000 7f1500d45700 1 -- 172.21.15.12:0/2711600357 wait complete. 2024-03-26T04:40:47.979 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial admin user... 2024-03-26T04:40:48.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:48 smithi012 ceph-mon[29863]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:48.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:48 smithi012 ceph-mon[29863]: mgrmap e12: y(active, since 2s) 2024-03-26T04:40:48.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:48 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:48.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:48 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:48.254 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:48 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:49.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:49 smithi012 ceph-mon[29863]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:49.562 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:49.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$diei2Ga9FNKNhoT7ntYjGO5hqear14vf2T3wtvV5ByuMgy7rfZrhW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711428048, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T04:40:49.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.358+0000 7fedeccbe700 1 Processor -- start 2024-03-26T04:40:49.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.358+0000 7fedeccbe700 1 -- start start 2024-03-26T04:40:49.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.358+0000 7fedeccbe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede8106a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:49.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.358+0000 7fedeccbe700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede8079710 con 0x7fede8104640 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede659c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede8106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede659c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede8106a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42384/0 (socket says 172.21.15.12:42384) 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede659c700 1 -- 172.21.15.12:0/979812963 learned_addr learned my addr 172.21.15.12:0/979812963 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede659c700 1 -- 172.21.15.12:0/979812963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede8106fb0 con 0x7fede8104640 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede659c700 1 --2- 172.21.15.12:0/979812963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede8106a70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098a0 tx=0x7fedd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2a3f89acbd3e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede559a700 1 -- 172.21.15.12:0/979812963 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedd8004160 con 0x7fede8104640 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede559a700 1 -- 172.21.15.12:0/979812963 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fedd80042c0 con 0x7fede8104640 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.359+0000 7fede559a700 1 -- 172.21.15.12:0/979812963 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedd8011850 con 0x7fede8104640 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 -- 172.21.15.12:0/979812963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 msgr2=0x7fede8106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 --2- 172.21.15.12:0/979812963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede8106a70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098a0 tx=0x7fedd8009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 -- 172.21.15.12:0/979812963 shutdown_connections 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 --2- 172.21.15.12:0/979812963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede8106a70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 -- 172.21.15.12:0/979812963 >> 172.21.15.12:0/979812963 conn(0x7fede81002e0 msgr2=0x7fede8102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:49.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 -- 172.21.15.12:0/979812963 shutdown_connections 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.360+0000 7fedeccbe700 1 -- 172.21.15.12:0/979812963 wait complete. 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fedeccbe700 1 Processor -- start 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fedeccbe700 1 -- start start 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fedeccbe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede81a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fedeccbe700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede81a0620 con 0x7fede8104640 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fede659c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede81a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fede659c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede81a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42400/0 (socket says 172.21.15.12:42400) 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.361+0000 7fede659c700 1 -- 172.21.15.12:0/1761921346 learned_addr learned my addr 172.21.15.12:0/1761921346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fede659c700 1 -- 172.21.15.12:0/1761921346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8009550 con 0x7fede8104640 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fede659c700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede81a00e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fedd80119a0 tx=0x7fedd8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedd8011e90 con 0x7fede8104640 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fedd80038b0 con 0x7fede8104640 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede81a0820 con 0x7fede8104640 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fedd8024e60 con 0x7fede8104640 2024-03-26T04:40:49.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.362+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede81a0cc0 con 0x7fede8104640 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.363+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) v1 ==== 45941+0+0 (secure 0 0 0) 0x7fedd801a4c0 con 0x7fede8104640 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.363+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede8199920 con 0x7fede8104640 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.363+0000 7fedd77fe700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fedd0038f30 0x7fedd003b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.363+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fedd804cf90 con 0x7fede8104640 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.363+0000 7fede5d9b700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fedd0038f30 0x7fedd003b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.364+0000 7fede5d9b700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fedd0038f30 0x7fedd003b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.369+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fedd802d3f0 con 0x7fede8104640 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.595+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fede80634d0 con 0x7fedd0038f30 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.841+0000 7fedd77fe700 1 -- 172.21.15.12:0/1761921346 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fede80634d0 con 0x7fedd0038f30 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.843+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fedd0038f30 msgr2=0x7fedd003b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.843+0000 7fedeccbe700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fedd0038f30 0x7fedd003b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.843+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 msgr2=0x7fede81a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.843+0000 7fedeccbe700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede81a00e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fedd80119a0 tx=0x7fedd8011b70 comp rx=0 tx=0).stop 2024-03-26T04:40:49.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.843+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 shutdown_connections 2024-03-26T04:40:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.843+0000 7fedeccbe700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fedd0038f30 0x7fedd003b3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.844+0000 7fedeccbe700 1 --2- 172.21.15.12:0/1761921346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fede8104640 0x7fede81a00e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.844+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 >> 172.21.15.12:0/1761921346 conn(0x7fede81002e0 msgr2=0x7fede81026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.844+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 shutdown_connections 2024-03-26T04:40:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:48.844+0000 7fedeccbe700 1 -- 172.21.15.12:0/1761921346 wait complete. 2024-03-26T04:40:49.573 INFO:teuthology.orchestra.run.smithi012.stdout:Fetching dashboard port number... 2024-03-26T04:40:50.896 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2485102614' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.386+0000 7f86f7488700 1 Processor -- start 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f7488700 1 -- start start 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f7488700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f0079b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f7488700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f007a110 con 0x7f86f007b6e0 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f5224700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f0079b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f5224700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f0079b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56460/0 (socket says 172.21.15.12:56460) 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f5224700 1 -- 172.21.15.12:0/4212190330 learned_addr learned my addr 172.21.15.12:0/4212190330 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:51.162 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f5224700 1 -- 172.21.15.12:0/4212190330 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f007a250 con 0x7f86f007b6e0 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.387+0000 7f86f5224700 1 --2- 172.21.15.12:0/4212190330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f0079b40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86e00098a0 tx=0x7f86e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f856ccd133ea39ee server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86e7fff700 1 -- 172.21.15.12:0/4212190330 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86e0004160 con 0x7f86f007b6e0 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86e7fff700 1 -- 172.21.15.12:0/4212190330 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f86e00042c0 con 0x7f86f007b6e0 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86e7fff700 1 -- 172.21.15.12:0/4212190330 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86e0011850 con 0x7f86f007b6e0 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 -- 172.21.15.12:0/4212190330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 msgr2=0x7f86f0079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 --2- 172.21.15.12:0/4212190330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f0079b40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86e00098a0 tx=0x7f86e0009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 -- 172.21.15.12:0/4212190330 shutdown_connections 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 --2- 172.21.15.12:0/4212190330 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f0079b40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 -- 172.21.15.12:0/4212190330 >> 172.21.15.12:0/4212190330 conn(0x7f86f0101360 msgr2=0x7f86f01037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 -- 172.21.15.12:0/4212190330 shutdown_connections 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.388+0000 7f86f7488700 1 -- 172.21.15.12:0/4212190330 wait complete. 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f7488700 1 Processor -- start 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f7488700 1 -- start start 2024-03-26T04:40:51.163 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f7488700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f019bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f7488700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f019c1e0 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f5224700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f019bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f5224700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f019bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56474/0 (socket says 172.21.15.12:56474) 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f5224700 1 -- 172.21.15.12:0/2485102614 learned_addr learned my addr 172.21.15.12:0/2485102614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.389+0000 7f86f5224700 1 -- 172.21.15.12:0/2485102614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0009550 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.390+0000 7f86f5224700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f019bca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86e00119a0 tx=0x7f86e0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.390+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86e0011e90 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.390+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f86e00038b0 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.390+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f019c3e0 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.390+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f86e0024e60 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.390+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f019c880 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.391+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) v1 ==== 45941+0+0 (secure 0 0 0) 0x7f86e001a480 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.391+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f010c160 con 0x7f86f007b6e0 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.391+0000 7f86e67fc700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f86dc038f90 0x7f86dc03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.391+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f86e004cf80 con 0x7f86f007b6e0 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.391+0000 7f86f4a23700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f86dc038f90 0x7f86dc03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.391+0000 7f86f4a23700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f86dc038f90 0x7f86dc03b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86ec006fd0 tx=0x7f86ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.396+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f86e0018a40 con 0x7f86f007b6e0 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.597+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f86f019f260 con 0x7f86f007b6e0 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.598+0000 7f86e67fc700 1 -- 172.21.15.12:0/2485102614 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f86e00212a0 con 0x7f86f007b6e0 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f86dc038f90 msgr2=0x7f86dc03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f86dc038f90 0x7f86dc03b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86ec006fd0 tx=0x7f86ec006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 msgr2=0x7f86f019bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f019bca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86e00119a0 tx=0x7f86e0011b70 comp rx=0 tx=0).stop 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 shutdown_connections 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f86dc038f90 0x7f86dc03b450 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 --2- 172.21.15.12:0/2485102614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f86f007b6e0 0x7f86f019bca0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 >> 172.21.15.12:0/2485102614 conn(0x7f86f0101360 msgr2=0x7f86f0101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:51.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 shutdown_connections 2024-03-26T04:40:51.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:50.600+0000 7f86f7488700 1 -- 172.21.15.12:0/2485102614 wait complete. 2024-03-26T04:40:51.169 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:40:51.169 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-03-26T04:40:51.175 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:40:51.175 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-03-26T04:40:51.176 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-03-26T04:40:51.176 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph Dashboard is now available at: 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout: URL: https://smithi012.front.sepia.ceph.com:8443/ 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout: User: admin 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout: Password: dh0827ve5p 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:51.178 INFO:teuthology.orchestra.run.smithi012.stdout:Saving cluster configuration to /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config directory 2024-03-26T04:40:51.179 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling autotune for osd_memory_target 2024-03-26T04:40:52.110 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:51 smithi012 ceph-mon[29863]: mgrmap e13: y(active, since 6s) 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f3d2f2700 1 Processor -- start 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f3d2f2700 1 -- start start 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f3d2f2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f38104af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f3d2f2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f38079690 con 0x7f4f381046d0 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f36ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f38104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f36ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f38104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56490/0 (socket says 172.21.15.12:56490) 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.612+0000 7f4f36ffd700 1 -- 172.21.15.12:0/1794941985 learned_addr learned my addr 172.21.15.12:0/1794941985 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:52.327 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f36ffd700 1 -- 172.21.15.12:0/1794941985 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f38105030 con 0x7f4f381046d0 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f36ffd700 1 --2- 172.21.15.12:0/1794941985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f38104af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f280098a0 tx=0x7f4f28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2015ed14fea8a9a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f35ffb700 1 -- 172.21.15.12:0/1794941985 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f28004160 con 0x7f4f381046d0 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f35ffb700 1 -- 172.21.15.12:0/1794941985 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4f280042c0 con 0x7f4f381046d0 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f35ffb700 1 -- 172.21.15.12:0/1794941985 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f28011850 con 0x7f4f381046d0 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/1794941985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 msgr2=0x7f4f38104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.613+0000 7f4f3d2f2700 1 --2- 172.21.15.12:0/1794941985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f38104af0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f280098a0 tx=0x7f4f28009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/1794941985 shutdown_connections 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 --2- 172.21.15.12:0/1794941985 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f38104af0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/1794941985 >> 172.21.15.12:0/1794941985 conn(0x7f4f381002d0 msgr2=0x7f4f381026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/1794941985 shutdown_connections 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/1794941985 wait complete. 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 Processor -- start 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.614+0000 7f4f3d2f2700 1 -- start start 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f3d2f2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f3819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:52.328 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f3d2f2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3819c1c0 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f36ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f3819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f36ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f3819bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56504/0 (socket says 172.21.15.12:56504) 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f36ffd700 1 -- 172.21.15.12:0/781008808 learned_addr learned my addr 172.21.15.12:0/781008808 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f36ffd700 1 -- 172.21.15.12:0/781008808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f28009550 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f36ffd700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f3819bc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f28006af0 tx=0x7f4f28011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f280036a0 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4f2801a430 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3819c3c0 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f2801ae60 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.615+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3819c860 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.616+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f4f28003800 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.616+0000 7f4f1ffff700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f20038f80 0x7f4f2003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.616+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38195950 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.616+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f2804d100 con 0x7f4f381046d0 2024-03-26T04:40:52.329 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.616+0000 7f4f367fc700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f20038f80 0x7f4f2003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.617+0000 7f4f367fc700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f20038f80 0x7f4f2003b440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c006fd0 tx=0x7f4f2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.621+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4f2802de30 con 0x7f4f381046d0 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.831+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f4f38105e20 con 0x7f4f381046d0 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.832+0000 7f4f1ffff700 1 -- 172.21.15.12:0/781008808 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4f280360c0 con 0x7f4f381046d0 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.833+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f20038f80 msgr2=0x7f4f2003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.833+0000 7f4f3d2f2700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f20038f80 0x7f4f2003b440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c006fd0 tx=0x7f4f2c006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.833+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 msgr2=0x7f4f3819bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.833+0000 7f4f3d2f2700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f3819bc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f28006af0 tx=0x7f4f28011a30 comp rx=0 tx=0).stop 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.834+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 shutdown_connections 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.834+0000 7f4f3d2f2700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f20038f80 0x7f4f2003b440 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.834+0000 7f4f3d2f2700 1 --2- 172.21.15.12:0/781008808 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f381046d0 0x7f4f3819bc80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.834+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 >> 172.21.15.12:0/781008808 conn(0x7f4f381002d0 msgr2=0x7f4f38192a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:52.330 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.834+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 shutdown_connections 2024-03-26T04:40:52.331 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:51.834+0000 7f4f3d2f2700 1 -- 172.21.15.12:0/781008808 wait complete. 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.743+0000 7f4f6698c700 1 Processor -- start 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.743+0000 7f4f6698c700 1 -- start start 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f6698c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f60106210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f6698c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f601067e0 con 0x7f4f60105df0 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f60106210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f60106210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56508/0 (socket says 172.21.15.12:56508) 2024-03-26T04:40:53.651 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5ffff700 1 -- 172.21.15.12:0/3745014989 learned_addr learned my addr 172.21.15.12:0/3745014989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5ffff700 1 -- 172.21.15.12:0/3745014989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f60106ff0 con 0x7f4f60105df0 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5ffff700 1 --2- 172.21.15.12:0/3745014989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f60106210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f500098a0 tx=0x7f4f50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e5838ebeaca5f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5effd700 1 -- 172.21.15.12:0/3745014989 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f50004160 con 0x7f4f60105df0 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5effd700 1 -- 172.21.15.12:0/3745014989 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4f500042c0 con 0x7f4f60105df0 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.744+0000 7f4f5effd700 1 -- 172.21.15.12:0/3745014989 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f50011850 con 0x7f4f60105df0 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 -- 172.21.15.12:0/3745014989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 msgr2=0x7f4f60106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 --2- 172.21.15.12:0/3745014989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f60106210 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f500098a0 tx=0x7f4f50009bb0 comp rx=0 tx=0).stop 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 -- 172.21.15.12:0/3745014989 shutdown_connections 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 --2- 172.21.15.12:0/3745014989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f60106210 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 -- 172.21.15.12:0/3745014989 >> 172.21.15.12:0/3745014989 conn(0x7f4f601013f0 msgr2=0x7f4f60103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 -- 172.21.15.12:0/3745014989 shutdown_connections 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.745+0000 7f4f6698c700 1 -- 172.21.15.12:0/3745014989 wait complete. 2024-03-26T04:40:53.652 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f6698c700 1 Processor -- start 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f6698c700 1 -- start start 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f6698c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f6007b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f6698c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6007b910 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f5ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f6007b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f5ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f6007b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56524/0 (socket says 172.21.15.12:56524) 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f5ffff700 1 -- 172.21.15.12:0/2264508589 learned_addr learned my addr 172.21.15.12:0/2264508589 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.746+0000 7f4f5ffff700 1 -- 172.21.15.12:0/2264508589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f50009550 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.747+0000 7f4f5ffff700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f6007b3d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f50006af0 tx=0x7f4f50011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.747+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f500036a0 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.747+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4f5001a430 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.747+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f60079b00 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.747+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f500193c0 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.747+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f60079fa0 con 0x7f4f60105df0 2024-03-26T04:40:53.653 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.748+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f4f50019520 con 0x7f4f60105df0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.748+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f600636b0 con 0x7f4f60105df0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.748+0000 7f4f5d7fa700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f48038fe0 0x7f4f4803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.748+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f5004d270 con 0x7f4f60105df0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.748+0000 7f4f5f7fe700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f48038fe0 0x7f4f4803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.748+0000 7f4f5f7fe700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f48038fe0 0x7f4f4803b4a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f54006fd0 tx=0x7f4f54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:52.753+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4f50023570 con 0x7f4f60105df0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.069+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4f6010a220 con 0x7f4f60105df0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.071+0000 7f4f5d7fa700 1 -- 172.21.15.12:0/2264508589 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4f50023390 con 0x7f4f60105df0 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f48038fe0 msgr2=0x7f4f4803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f48038fe0 0x7f4f4803b4a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f54006fd0 tx=0x7f4f54006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 msgr2=0x7f4f6007b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f6007b3d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f50006af0 tx=0x7f4f50011a50 comp rx=0 tx=0).stop 2024-03-26T04:40:53.654 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 shutdown_connections 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f48038fe0 0x7f4f4803b4a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 --2- 172.21.15.12:0/2264508589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f60105df0 0x7f4f6007b3d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.073+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 >> 172.21.15.12:0/2264508589 conn(0x7f4f601013f0 msgr2=0x7f4f60101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.074+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 shutdown_connections 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-03-26T04:40:53.074+0000 7f4f6698c700 1 -- 172.21.15.12:0/2264508589 wait complete. 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T04:40:53.655 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout: ceph telemetry on 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout:For more information see: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:40:53.656 INFO:teuthology.orchestra.run.smithi012.stdout:Bootstrap complete. 2024-03-26T04:40:53.688 INFO:tasks.cephadm:Fetching config... 2024-03-26T04:40:53.689 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:40:53.689 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T04:40:53.704 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T04:40:53.704 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:40:53.704 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T04:40:53.758 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T04:40:53.758 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:40:53.759 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T04:40:53.826 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T04:40:53.826 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:40:53.826 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T04:40:53.881 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T04:40:53.881 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsO79fSaetRlmZIj6pHsVEuKnWb4jxGeOHfz1uRW7cVJHQtsgmlWxzDjUSUNMieIsHH8wdUTSAOFLTPqgCMp6lCKhja1oYNR7b1i+/fKvUVycgahyz2fzUNRBtpIVPccGEZweqixj6GjmjR+6qgOaKuZlpLatKnBFRTOJlrJjC6W/ozAluhGwm4EypIjWMtQwJJNr+PlTfpg47g7GveUXrgonLNI40fT8v2SNsdb7lEqlWqVFVNKKzPhi99VuWK+N7plWUUd49DhSuTOBLoJKMfLa4nRe805LK5HETVDlvMGJ9DMWzs6R9+dvzIh0dNOVE4TPHnkym8bAiB6eN14Ci5FntsNNfIX68TD7VMDIqwbT2WArNnutRJWh2dPphAG4nt/wXMrrQPxcHGPsfgIhlz7Q8Hzaeh2FhNYFabx8dNGFwOeZASGUFoSTxDA5GEg1QDldGZmCcgckw0VQXJkgc1K0y58eZEnHCwfbaHG6mD9ioBEHMhoYnHbpPbpHfZhs= ceph-c3b00422-eb2a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:40:53.962 INFO:teuthology.orchestra.run.smithi012.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsO79fSaetRlmZIj6pHsVEuKnWb4jxGeOHfz1uRW7cVJHQtsgmlWxzDjUSUNMieIsHH8wdUTSAOFLTPqgCMp6lCKhja1oYNR7b1i+/fKvUVycgahyz2fzUNRBtpIVPccGEZweqixj6GjmjR+6qgOaKuZlpLatKnBFRTOJlrJjC6W/ozAluhGwm4EypIjWMtQwJJNr+PlTfpg47g7GveUXrgonLNI40fT8v2SNsdb7lEqlWqVFVNKKzPhi99VuWK+N7plWUUd49DhSuTOBLoJKMfLa4nRe805LK5HETVDlvMGJ9DMWzs6R9+dvzIh0dNOVE4TPHnkym8bAiB6eN14Ci5FntsNNfIX68TD7VMDIqwbT2WArNnutRJWh2dPphAG4nt/wXMrrQPxcHGPsfgIhlz7Q8Hzaeh2FhNYFabx8dNGFwOeZASGUFoSTxDA5GEg1QDldGZmCcgckw0VQXJkgc1K0y58eZEnHCwfbaHG6mD9ioBEHMhoYnHbpPbpHfZhs= ceph-c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:40:53.976 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsO79fSaetRlmZIj6pHsVEuKnWb4jxGeOHfz1uRW7cVJHQtsgmlWxzDjUSUNMieIsHH8wdUTSAOFLTPqgCMp6lCKhja1oYNR7b1i+/fKvUVycgahyz2fzUNRBtpIVPccGEZweqixj6GjmjR+6qgOaKuZlpLatKnBFRTOJlrJjC6W/ozAluhGwm4EypIjWMtQwJJNr+PlTfpg47g7GveUXrgonLNI40fT8v2SNsdb7lEqlWqVFVNKKzPhi99VuWK+N7plWUUd49DhSuTOBLoJKMfLa4nRe805LK5HETVDlvMGJ9DMWzs6R9+dvzIh0dNOVE4TPHnkym8bAiB6eN14Ci5FntsNNfIX68TD7VMDIqwbT2WArNnutRJWh2dPphAG4nt/wXMrrQPxcHGPsfgIhlz7Q8Hzaeh2FhNYFabx8dNGFwOeZASGUFoSTxDA5GEg1QDldGZmCcgckw0VQXJkgc1K0y58eZEnHCwfbaHG6mD9ioBEHMhoYnHbpPbpHfZhs= ceph-c3b00422-eb2a-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:40:54.025 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsO79fSaetRlmZIj6pHsVEuKnWb4jxGeOHfz1uRW7cVJHQtsgmlWxzDjUSUNMieIsHH8wdUTSAOFLTPqgCMp6lCKhja1oYNR7b1i+/fKvUVycgahyz2fzUNRBtpIVPccGEZweqixj6GjmjR+6qgOaKuZlpLatKnBFRTOJlrJjC6W/ozAluhGwm4EypIjWMtQwJJNr+PlTfpg47g7GveUXrgonLNI40fT8v2SNsdb7lEqlWqVFVNKKzPhi99VuWK+N7plWUUd49DhSuTOBLoJKMfLa4nRe805LK5HETVDlvMGJ9DMWzs6R9+dvzIh0dNOVE4TPHnkym8bAiB6eN14Ci5FntsNNfIX68TD7VMDIqwbT2WArNnutRJWh2dPphAG4nt/wXMrrQPxcHGPsfgIhlz7Q8Hzaeh2FhNYFabx8dNGFwOeZASGUFoSTxDA5GEg1QDldGZmCcgckw0VQXJkgc1K0y58eZEnHCwfbaHG6mD9ioBEHMhoYnHbpPbpHfZhs= ceph-c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:40:54.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T04:40:54.281 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:54.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2264508589' entity='client.admin' 2024-03-26T04:40:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 -- 172.21.15.12:0/3973563727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 msgr2=0x7f6760101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 --2- 172.21.15.12:0/3973563727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760101ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f675c009910 tx=0x7f675c009c20 comp rx=0 tx=0).stop 2024-03-26T04:40:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 -- 172.21.15.12:0/3973563727 shutdown_connections 2024-03-26T04:40:56.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 --2- 172.21.15.12:0/3973563727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760101ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 -- 172.21.15.12:0/3973563727 >> 172.21.15.12:0/3973563727 conn(0x7f67600fd050 msgr2=0x7f67600ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 -- 172.21.15.12:0/3973563727 shutdown_connections 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 -- 172.21.15.12:0/3973563727 wait complete. 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.484+0000 7f67674a6700 1 Processor -- start 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67674a6700 1 -- start start 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67674a6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760075d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67674a6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6760076280 con 0x7f6760101ab0 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f6765242700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760075d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f6765242700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760075d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56542/0 (socket says 172.21.15.12:56542) 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f6765242700 1 -- 172.21.15.12:0/2389391029 learned_addr learned my addr 172.21.15.12:0/2389391029 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:56.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f6765242700 1 -- 172.21.15.12:0/2389391029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0095f0 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f6765242700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760075d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f675c005660 tx=0x7f675c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c00b960 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f675c00f460 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6760074470 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c017440 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.485+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6760074910 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.487+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f675c0176b0 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.487+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67600636b0 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.487+0000 7f67527fc700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f674c038fe0 0x7f674c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.487+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f675c04d4d0 con 0x7f6760101ab0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.487+0000 7f6764a41700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f674c038fe0 0x7f674c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.487+0000 7f6764a41700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f674c038fe0 0x7f674c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6754006fd0 tx=0x7f6754006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:56.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.492+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f675c015a40 con 0x7f6760101ab0 2024-03-26T04:40:56.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.716+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f6760106db0 con 0x7f6760101ab0 2024-03-26T04:40:56.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.719+0000 7f67527fc700 1 -- 172.21.15.12:0/2389391029 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f675c027030 con 0x7f6760101ab0 2024-03-26T04:40:56.719 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:56.719 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:56.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi012", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:56.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:56.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:56.720 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f674c038fe0 msgr2=0x7f674c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f674c038fe0 0x7f674c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6754006fd0 tx=0x7f6754006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:56.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 msgr2=0x7f6760075d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760075d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f675c005660 tx=0x7f675c005740 comp rx=0 tx=0).stop 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 shutdown_connections 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f674c038fe0 0x7f674c03b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 --2- 172.21.15.12:0/2389391029 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6760101ab0 0x7f6760075d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 >> 172.21.15.12:0/2389391029 conn(0x7f67600fd050 msgr2=0x7f67600fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 shutdown_connections 2024-03-26T04:40:56.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:56.722+0000 7f67674a6700 1 -- 172.21.15.12:0/2389391029 wait complete. 2024-03-26T04:40:57.590 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T04:40:57.590 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T04:40:57.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2389391029' entity='client.admin' 2024-03-26T04:40:57.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:57 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:57.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:40:57 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:40:58.262 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:40:59.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 -- 172.21.15.12:0/3593899031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 msgr2=0x7f9afc100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 --2- 172.21.15.12:0/3593899031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc100df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009910 tx=0x7f9af8009c20 comp rx=0 tx=0).stop 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 -- 172.21.15.12:0/3593899031 shutdown_connections 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 --2- 172.21.15.12:0/3593899031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc100df0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 -- 172.21.15.12:0/3593899031 >> 172.21.15.12:0/3593899031 conn(0x7f9afc0fbf90 msgr2=0x7f9afc0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 -- 172.21.15.12:0/3593899031 shutdown_connections 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.604+0000 7f9b047bf700 1 -- 172.21.15.12:0/3593899031 wait complete. 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b047bf700 1 Processor -- start 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b047bf700 1 -- start start 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b047bf700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc1965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b047bf700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc196b00 con 0x7f9afc1009d0 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b0255b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc1965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b0255b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc1965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38166/0 (socket says 172.21.15.12:38166) 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b0255b700 1 -- 172.21.15.12:0/2502865476 learned_addr learned my addr 172.21.15.12:0/2502865476 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b0255b700 1 -- 172.21.15.12:0/2502865476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af80095f0 con 0x7f9afc1009d0 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.605+0000 7f9b0255b700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc1965c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005660 tx=0x7f9af8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.606+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9af800b960 con 0x7f9afc1009d0 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.606+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9af800f460 con 0x7f9afc1009d0 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.606+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc196d00 con 0x7f9afc1009d0 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.606+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9af8017440 con 0x7f9afc1009d0 2024-03-26T04:40:59.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.606+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc1971a0 con 0x7f9afc1009d0 2024-03-26T04:40:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.607+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f9af800bac0 con 0x7f9afc1009d0 2024-03-26T04:40:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.607+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc190260 con 0x7f9afc1009d0 2024-03-26T04:40:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.607+0000 7f9af37fe700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f9ae8038f80 0x7f9ae803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:40:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.607+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9af804d1d0 con 0x7f9afc1009d0 2024-03-26T04:40:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.607+0000 7f9b01d5a700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f9ae8038f80 0x7f9ae803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:40:59.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.607+0000 7f9b01d5a700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f9ae8038f80 0x7f9ae803b440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9aec006fd0 tx=0x7f9aec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:40:59.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.612+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9af800f5d0 con 0x7f9afc1009d0 2024-03-26T04:40:59.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.819+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f9afc105f00 con 0x7f9ae8038f80 2024-03-26T04:40:59.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.822+0000 7f9af37fe700 1 -- 172.21.15.12:0/2502865476 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9afc105f00 con 0x7f9ae8038f80 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f9ae8038f80 msgr2=0x7f9ae803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f9ae8038f80 0x7f9ae803b440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9aec006fd0 tx=0x7f9aec006e40 comp rx=0 tx=0).stop 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 msgr2=0x7f9afc1965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc1965c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9af8005660 tx=0x7f9af8005740 comp rx=0 tx=0).stop 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 shutdown_connections 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f9ae8038f80 0x7f9ae803b440 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 --2- 172.21.15.12:0/2502865476 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9afc1009d0 0x7f9afc1965c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 >> 172.21.15.12:0/2502865476 conn(0x7f9afc0fbf90 msgr2=0x7f9afc06a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 shutdown_connections 2024-03-26T04:40:59.825 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:40:59.824+0000 7f9b047bf700 1 -- 172.21.15.12:0/2502865476 wait complete. 2024-03-26T04:41:00.541 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-03-26T04:41:00.541 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:41:00.542 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:41:00.559 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:41:00.560 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:00.616 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-03-26T04:41:00.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch host add smithi186 2024-03-26T04:41:00.858 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:00.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:00.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:00.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:00.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:00.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:00.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/2446019561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 msgr2=0x7f7b88106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 --2- 172.21.15.12:0/2446019561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88106d30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009910 tx=0x7f7b84009c20 comp rx=0 tx=0).stop 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/2446019561 shutdown_connections 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 --2- 172.21.15.12:0/2446019561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88106d30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/2446019561 >> 172.21.15.12:0/2446019561 conn(0x7f7b880fc440 msgr2=0x7f7b880fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/2446019561 shutdown_connections 2024-03-26T04:41:02.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/2446019561 wait complete. 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 Processor -- start 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- start start 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.278+0000 7f7b8f2cb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8806ae10 con 0x7f7b88106950 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8d067700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8d067700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38192/0 (socket says 172.21.15.12:38192) 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8d067700 1 -- 172.21.15.12:0/4093714204 learned_addr learned my addr 172.21.15.12:0/4093714204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8d067700 1 -- 172.21.15.12:0/4093714204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840095f0 con 0x7f7b88106950 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8d067700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88197730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7b84005660 tx=0x7f7b84005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b8401d070 con 0x7f7b88106950 2024-03-26T04:41:02.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7b8400b750 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b88197d30 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b840037b0 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.279+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b881981b0 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.280+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f7b84003a20 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.280+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b88104c00 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.280+0000 7f7b7e7fc700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f7b74038fe0 0x7f7b7403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.280+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7b8404cd70 con 0x7f7b88106950 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.280+0000 7f7b8c866700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f7b74038fe0 0x7f7b7403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:02.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.281+0000 7f7b8c866700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f7b74038fe0 0x7f7b7403b4a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006fd0 tx=0x7f7b78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:02.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.285+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7b84051070 con 0x7f7b88106950 2024-03-26T04:41:02.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:02.494+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b88198400 con 0x7f7b74038fe0 2024-03-26T04:41:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:02 smithi012 ceph-mon[29863]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:03 smithi012 ceph-mon[29863]: Deploying cephadm binary to smithi186 2024-03-26T04:41:05.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:05 smithi012 ceph-mon[29863]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:06.936 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.935+0000 7f7b7e7fc700 1 -- 172.21.15.12:0/4093714204 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7b88198400 con 0x7f7b74038fe0 2024-03-26T04:41:06.936 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-03-26T04:41:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f7b74038fe0 msgr2=0x7f7b7403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f7b74038fe0 0x7f7b7403b4a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7b78006fd0 tx=0x7f7b78006e40 comp rx=0 tx=0).stop 2024-03-26T04:41:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 msgr2=0x7f7b88197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88197730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7b84005660 tx=0x7f7b84005740 comp rx=0 tx=0).stop 2024-03-26T04:41:06.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 shutdown_connections 2024-03-26T04:41:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f7b74038fe0 0x7f7b7403b4a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 --2- 172.21.15.12:0/4093714204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7b88106950 0x7f7b88197730 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 >> 172.21.15.12:0/4093714204 conn(0x7f7b880fc440 msgr2=0x7f7b88108810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 shutdown_connections 2024-03-26T04:41:06.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:06.937+0000 7f7b8f2cb700 1 -- 172.21.15.12:0/4093714204 wait complete. 2024-03-26T04:41:07.413 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:41:07.654 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:07.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:07 smithi012 ceph-mon[29863]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:07.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:07 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:07.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:07 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:07.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:07 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:08 smithi012 ceph-mon[29863]: Added host smithi186 2024-03-26T04:41:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 -- 172.21.15.12:0/613216293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 msgr2=0x7fb41c102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:09.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 --2- 172.21.15.12:0/613216293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c102fe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb418009910 tx=0x7fb418009c20 comp rx=0 tx=0).stop 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 -- 172.21.15.12:0/613216293 shutdown_connections 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 --2- 172.21.15.12:0/613216293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c102fe0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 -- 172.21.15.12:0/613216293 >> 172.21.15.12:0/613216293 conn(0x7fb41c0fc430 msgr2=0x7fb41c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 -- 172.21.15.12:0/613216293 shutdown_connections 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 -- 172.21.15.12:0/613216293 wait complete. 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.113+0000 7fb4238d8700 1 Processor -- start 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4238d8700 1 -- start start 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4238d8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4238d8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c19cb40 con 0x7fb41c102c00 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb421674700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb421674700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c19c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50898/0 (socket says 172.21.15.12:50898) 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb421674700 1 -- 172.21.15.12:0/1867198996 learned_addr learned my addr 172.21.15.12:0/1867198996 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:41:09.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb421674700 1 -- 172.21.15.12:0/1867198996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180095f0 con 0x7fb41c102c00 2024-03-26T04:41:09.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb421674700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c19c600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4180055f0 tx=0x7fb4180056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb41801d070 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb41800b750 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c19cd40 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4180037b0 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.114+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c19d160 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.115+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7fb418004390 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.115+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c06a5e0 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.116+0000 7fb4127fc700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb408038f80 0x7fb40803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.116+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb41804ce10 con 0x7fb41c102c00 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.116+0000 7fb420e73700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb408038f80 0x7fb40803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:09.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.116+0000 7fb420e73700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb408038f80 0x7fb40803b440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb40c006fd0 tx=0x7fb40c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:09.121 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.121+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb41800bb80 con 0x7fb41c102c00 2024-03-26T04:41:09.327 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.327+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb41c10b3f0 con 0x7fb408038f80 2024-03-26T04:41:09.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.328+0000 7fb4127fc700 1 -- 172.21.15.12:0/1867198996 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fb41c10b3f0 con 0x7fb408038f80 2024-03-26T04:41:09.328 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:41:09.328 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-03-26T04:41:09.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb408038f80 msgr2=0x7fb40803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:09.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb408038f80 0x7fb40803b440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb40c006fd0 tx=0x7fb40c006e40 comp rx=0 tx=0).stop 2024-03-26T04:41:09.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 msgr2=0x7fb41c19c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:09.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c19c600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4180055f0 tx=0x7fb4180056d0 comp rx=0 tx=0).stop 2024-03-26T04:41:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 shutdown_connections 2024-03-26T04:41:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb408038f80 0x7fb40803b440 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 --2- 172.21.15.12:0/1867198996 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb41c102c00 0x7fb41c19c600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 >> 172.21.15.12:0/1867198996 conn(0x7fb41c0fc430 msgr2=0x7fb41c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 shutdown_connections 2024-03-26T04:41:09.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:09.330+0000 7fb4238d8700 1 -- 172.21.15.12:0/1867198996 wait complete. 2024-03-26T04:41:09.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:09 smithi012 ceph-mon[29863]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:09.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:09.940 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T04:41:09.940 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T04:41:10.177 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:10.827 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:10 smithi012 ceph-mon[29863]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 -- 172.21.15.12:0/1200343248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 msgr2=0x7fa5dc106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 --2- 172.21.15.12:0/1200343248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc106d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009910 tx=0x7fa5cc009c20 comp rx=0 tx=0).stop 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 -- 172.21.15.12:0/1200343248 shutdown_connections 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 --2- 172.21.15.12:0/1200343248 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc106d40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 -- 172.21.15.12:0/1200343248 >> 172.21.15.12:0/1200343248 conn(0x7fa5dc0fc450 msgr2=0x7fa5dc0fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 -- 172.21.15.12:0/1200343248 shutdown_connections 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.562+0000 7fa5e10b6700 1 -- 172.21.15.12:0/1200343248 wait complete. 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.563+0000 7fa5e10b6700 1 Processor -- start 2024-03-26T04:41:11.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.563+0000 7fa5e10b6700 1 -- start start 2024-03-26T04:41:11.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.563+0000 7fa5e10b6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:11.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.563+0000 7fa5e10b6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5dc06ae20 con 0x7fa5dc106960 2024-03-26T04:41:11.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5dad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:11.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5dad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50918/0 (socket says 172.21.15.12:50918) 2024-03-26T04:41:11.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5dad9d700 1 -- 172.21.15.12:0/677730465 learned_addr learned my addr 172.21.15.12:0/677730465 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:41:11.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5dad9d700 1 -- 172.21.15.12:0/677730465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0095f0 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5dad9d700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc197870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5cc01d070 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa5cc00b750 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5cc003810 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5dc197db0 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.564+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5dc198230 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.565+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7fa5cc003a00 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.565+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5dc104c10 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.565+0000 7fa5c3fff700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa5c4038fe0 0x7fa5c403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.565+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa5cc04cf30 con 0x7fa5dc106960 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.565+0000 7fa5da59c700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa5c4038fe0 0x7fa5c403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:11.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.566+0000 7fa5da59c700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa5c4038fe0 0x7fa5c403b4a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0006fd0 tx=0x7fa5d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:11.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.570+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa5cc029360 con 0x7fa5dc106960 2024-03-26T04:41:11.782 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:11.781+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa5dc1020a0 con 0x7fa5dc106960 2024-03-26T04:41:11.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:11 smithi012 ceph-mon[29863]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:12.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.610+0000 7fa5c3fff700 1 -- 172.21.15.12:0/677730465 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa5cc026070 con 0x7fa5dc106960 2024-03-26T04:41:12.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa5c4038fe0 msgr2=0x7fa5c403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa5c4038fe0 0x7fa5c403b4a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0006fd0 tx=0x7fa5d0006e40 comp rx=0 tx=0).stop 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 msgr2=0x7fa5dc197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc197870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc005710 comp rx=0 tx=0).stop 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 shutdown_connections 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa5c4038fe0 0x7fa5c403b4a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 --2- 172.21.15.12:0/677730465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5dc106960 0x7fa5dc197870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 >> 172.21.15.12:0/677730465 conn(0x7fa5dc0fc450 msgr2=0x7fa5dc108830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 shutdown_connections 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:12.613+0000 7fa5e10b6700 1 -- 172.21.15.12:0/677730465 wait complete. 2024-03-26T04:41:12.614 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/677730465' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:13.246 INFO:tasks.cephadm:Adding mon.a on smithi012 2024-03-26T04:41:13.247 INFO:tasks.cephadm:Adding mon.c on smithi012 2024-03-26T04:41:13.247 INFO:tasks.cephadm:Adding mon.b on smithi186 2024-03-26T04:41:13.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi012:172.21.15.12=a;smithi012:[v2:172.21.15.12:3301,v1:172.21.15.12:6790]=c;smithi186:172.21.15.186=b' 2024-03-26T04:41:13.482 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:41:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/677730465' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:41:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:13.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 -- 172.21.15.186:0/3403541984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 msgr2=0x7f1074106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 --2- 172.21.15.186:0/3403541984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f1074106d20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f106c00b0a0 tx=0x7f106c00b3b0 comp rx=0 tx=0).stop 2024-03-26T04:41:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 -- 172.21.15.186:0/3403541984 shutdown_connections 2024-03-26T04:41:14.795 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 --2- 172.21.15.186:0/3403541984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f1074106d20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 -- 172.21.15.186:0/3403541984 >> 172.21.15.186:0/3403541984 conn(0x7f10740fc410 msgr2=0x7f10740fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 -- 172.21.15.186:0/3403541984 shutdown_connections 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 -- 172.21.15.186:0/3403541984 wait complete. 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 Processor -- start 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.794+0000 7f107c7ff700 1 -- start start 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107c7ff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f107419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107c7ff700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419cb60 con 0x7f1074106940 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107a59b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f107419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107a59b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f107419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.186:60714/0 (socket says 172.21.15.186:60714) 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107a59b700 1 -- 172.21.15.186:0/2415710759 learned_addr learned my addr 172.21.15.186:0/2415710759 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107a59b700 1 -- 172.21.15.186:0/2415710759 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c009d00 con 0x7f1074106940 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107a59b700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f107419c620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f106c009610 tx=0x7f106c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f106c00e040 con 0x7f1074106940 2024-03-26T04:41:14.796 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f106c003c10 con 0x7f1074106940 2024-03-26T04:41:14.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107c7ff700 1 -- 172.21.15.186:0/2415710759 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107419cd60 con 0x7f1074106940 2024-03-26T04:41:14.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f106c01d470 con 0x7f1074106940 2024-03-26T04:41:14.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.795+0000 7f107c7ff700 1 -- 172.21.15.186:0/2415710759 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107419d1e0 con 0x7f1074106940 2024-03-26T04:41:14.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.796+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1054005120 con 0x7f1074106940 2024-03-26T04:41:14.797 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.796+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f106c019070 con 0x7f1074106940 2024-03-26T04:41:14.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.797+0000 7f10737fe700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f105c038f80 0x7f105c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:14.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.797+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f106c04cff0 con 0x7f1074106940 2024-03-26T04:41:14.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.797+0000 7f1079d9a700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f105c038f80 0x7f105c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:14.798 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.797+0000 7f1079d9a700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f105c038f80 0x7f105c03b440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1068006fd0 tx=0x7f1068006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:14.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:14.802+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f106c004040 con 0x7f1074106940 2024-03-26T04:41:14.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:14 smithi012 ceph-mon[29863]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:14.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:14 smithi012 ceph-mon[29863]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:14.824 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:14 smithi012 ceph-mon[29863]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:14.825 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:14 smithi012 ceph-mon[29863]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:15.008 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.008+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi012:[v2:172.21.15.12:3301,v1:172.21.15.12:6790]=c;smithi186:172.21.15.186=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f1054000c90 con 0x7f105c038f80 2024-03-26T04:41:15.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.016+0000 7f10737fe700 1 -- 172.21.15.186:0/2415710759 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1054000c90 con 0x7f105c038f80 2024-03-26T04:41:15.016 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-03-26T04:41:15.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f105c038f80 msgr2=0x7f105c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:15.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f105c038f80 0x7f105c03b440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1068006fd0 tx=0x7f1068006e40 comp rx=0 tx=0).stop 2024-03-26T04:41:15.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 msgr2=0x7f107419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:15.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f107419c620 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f106c009610 tx=0x7f106c00bd40 comp rx=0 tx=0).stop 2024-03-26T04:41:15.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 shutdown_connections 2024-03-26T04:41:15.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f105c038f80 0x7f105c03b440 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:15.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 --2- 172.21.15.186:0/2415710759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1074106940 0x7f107419c620 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:15.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 >> 172.21.15.186:0/2415710759 conn(0x7f10740fc410 msgr2=0x7f107406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:15.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 shutdown_connections 2024-03-26T04:41:15.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:15.018+0000 7f10717fa700 1 -- 172.21.15.186:0/2415710759 wait complete. 2024-03-26T04:41:15.550 DEBUG:teuthology.orchestra.run.smithi012:mon.c> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T04:41:15.552 DEBUG:teuthology.orchestra.run.smithi186:mon.b> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T04:41:15.554 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:41:15.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:41:16.353 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi012:[v2:172.21.15.12:3301,v1:172.21.15.12:6790]=c;smithi186:172.21.15.186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi012:[v2:172.21.15.12:3301,v1:172.21.15.12:6790]=c;smithi186:172.21.15.186=b;count:3 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:16.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:16.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:16.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:16 smithi012 ceph-mon[29863]: Deploying daemon mon.b on smithi186 2024-03-26T04:41:17.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:17 smithi012 ceph-mon[29863]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:19.080 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 -- 172.21.15.186:0/2449651465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 msgr2=0x7f8254106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 --2- 172.21.15.186:0/2449651465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f8254106e40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8248009910 tx=0x7f8248009c20 comp rx=0 tx=0).stop 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 -- 172.21.15.186:0/2449651465 shutdown_connections 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 --2- 172.21.15.186:0/2449651465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f8254106e40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 -- 172.21.15.186:0/2449651465 >> 172.21.15.186:0/2449651465 conn(0x7f82540fbbd0 msgr2=0x7f82540fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 -- 172.21.15.186:0/2449651465 shutdown_connections 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.078+0000 7f82597dc700 1 -- 172.21.15.186:0/2449651465 wait complete. 2024-03-26T04:41:19.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.079+0000 7f82597dc700 1 Processor -- start 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.079+0000 7f82597dc700 1 -- start start 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.079+0000 7f82597dc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f825419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.079+0000 7f82597dc700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825419cbd0 con 0x7f8254106a60 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f8252ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f825419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f8252ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f825419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.186:60742/0 (socket says 172.21.15.186:60742) 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f8252ffd700 1 -- 172.21.15.186:0/402917130 learned_addr learned my addr 172.21.15.186:0/402917130 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f8252ffd700 1 -- 172.21.15.186:0/402917130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480095f0 con 0x7f8254106a60 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f8252ffd700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f825419c690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8248005660 tx=0x7f8248005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f824801d070 con 0x7f8254106a60 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825419ce30 con 0x7f8254106a60 2024-03-26T04:41:19.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.080+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f824800b6f0 con 0x7f8254106a60 2024-03-26T04:41:19.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.081+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82480037f0 con 0x7f8254106a60 2024-03-26T04:41:19.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.081+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8254196760 con 0x7f8254106a60 2024-03-26T04:41:19.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.081+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82541032b0 con 0x7f8254106a60 2024-03-26T04:41:19.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.082+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f8248004390 con 0x7f8254106a60 2024-03-26T04:41:19.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.082+0000 7f823bfff700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f823c038f80 0x7f823c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:19.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.082+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f824804cc40 con 0x7f8254106a60 2024-03-26T04:41:19.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.082+0000 7f82527fc700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f823c038f80 0x7f823c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:19.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.082+0000 7f82527fc700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f823c038f80 0x7f823c03b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8244006fd0 tx=0x7f8244006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:19.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.087+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f824800bb20 con 0x7f8254106a60 2024-03-26T04:41:19.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:18 smithi012 ceph-mon[29863]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:19.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.381+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8254107550 con 0x7f8254106a60 2024-03-26T04:41:19.381 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.381+0000 7f823bfff700 1 -- 172.21.15.186:0/402917130 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f8254107550 con 0x7f8254106a60 2024-03-26T04:41:19.382 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:41:19.382 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"c3b00422-eb2a-11ee-95d0-87774f69a715","modified":"2024-03-26T04:39:41.201484Z","created":"2024-03-26T04:39:41.201484Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.383+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f823c038f80 msgr2=0x7f823c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.383+0000 7f82597dc700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f823c038f80 0x7f823c03b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8244006fd0 tx=0x7f8244006e40 comp rx=0 tx=0).stop 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.383+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 msgr2=0x7f825419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.383+0000 7f82597dc700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f825419c690 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8248005660 tx=0x7f8248005740 comp rx=0 tx=0).stop 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.383+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 shutdown_connections 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.383+0000 7f82597dc700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f823c038f80 0x7f823c03b440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.384+0000 7f82597dc700 1 --2- 172.21.15.186:0/402917130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8254106a60 0x7f825419c690 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.384+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 >> 172.21.15.186:0/402917130 conn(0x7f82540fbbd0 msgr2=0x7f8254109920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.384+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 shutdown_connections 2024-03-26T04:41:19.384 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:19.384+0000 7f82597dc700 1 -- 172.21.15.186:0/402917130 wait complete. 2024-03-26T04:41:19.385 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-03-26T04:41:20.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/402917130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 podman[33419]: 2024-03-26 04:41:20.074661215 +0000 UTC m=+0.018086375 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:41:20.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 podman[33419]: 2024-03-26 04:41:20.357133011 +0000 UTC m=+0.300558158 container init 8fc10543afb1e297b001b7a48272a011dcb7dc8f640043cb7650a52a639fcdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T04:41:20.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 podman[33419]: 2024-03-26 04:41:20.360037027 +0000 UTC m=+0.303462167 container start 8fc10543afb1e297b001b7a48272a011dcb7dc8f640043cb7650a52a639fcdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:41:20.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pidfile_write: ignore empty --pid-file 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: load: jerasure load: lrc 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Git sha 0 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: DB SUMMARY 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: DB Session ID: E55W7TX665NB0A9Q425Y 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:41:20.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.env: 0x556d9f4ccae0 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:41:20.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.info_log: 0x556da0af5920 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.statistics: (nil) 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.use_fsync: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:41:20.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.db_log_dir: 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.wal_dir: 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.write_buffer_manager: 0x556da0ac81e0 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:41:20.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.unordered_write: 0 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.row_cache: None 2024-03-26T04:41:20.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.wal_filter: None 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.wal_compression: 0 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:41:20.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:41:20.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_open_files: -1 2024-03-26T04:41:20.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Compression algorithms supported: 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kZSTD supported: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:41:20.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.merge_operator: 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_filter: None 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556da0af5a20) 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:41:20.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_type: 0 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_index_type: 0 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_shortening: 1 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: checksum: 4 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: no_block_cache: 0 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache: 0x556da0af9090 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_options: 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: capacity : 536870912 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_shard_bits : 4 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: strict_capacity_limit : 0 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_compressed: (nil) 2024-03-26T04:41:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: persistent_cache: (nil) 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size: 4096 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size_deviation: 10 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_restart_interval: 16 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_block_restart_interval: 1 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: metadata_block_size: 4096 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: partition_filters: 0 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: use_delta_encoding: 1 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: filter_policy: bloomfilter 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: whole_key_filtering: 1 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: verify_compression: 0 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: format_version: 5 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_index_compression: 1 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_align: 0 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: max_auto_readahead_size: 262144 2024-03-26T04:41:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: prepopulate_block_cache: 0 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression: NoCompression 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.num_levels: 7 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:41:20.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:41:20.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:41:20.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:41:20.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:41:20.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.ttl: 2592000 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:41:20.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3e4d4f9f-8a70-4225-9df3-af3bb8f2d1c6 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428080389534, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428080389962, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711428080, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3e4d4f9f-8a70-4225-9df3-af3bb8f2d1c6", "db_session_id": "E55W7TX665NB0A9Q425Y", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428080390043, "job": 1, "event": "recovery_finished"} 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556da0bea000 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: DB pointer 0x556da0bd4000 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:41:20.734 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** DB Stats ** 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:41:20.735 INFO:journalctl@ceph.mon.b.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:41:20.736 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: Block cache BinnedLRUCache@0x556da0af9090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: using public_addr v2:172.21.15.186:0/0 -> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: starting mon.b rank -1 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(???) e0 preinit fsid c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: e1 2024-03-26T04:41:20.737 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout: legacy client fscid: -1 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout: No filesystems configured 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Added host smithi186 2024-03-26T04:41:20.738 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/677730465' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/677730465' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:41:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi012:[v2:172.21.15.12:3301,v1:172.21.15.12:6790]=c;smithi186:172.21.15.186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi012:[v2:172.21.15.12:3301,v1:172.21.15.12:6790]=c;smithi186:172.21.15.186=b;count:3 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:20.740 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: Deploying daemon mon.b on smithi186 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/402917130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 ceph-mon[33438]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 bash[33419]: 8fc10543afb1e297b001b7a48272a011dcb7dc8f640043cb7650a52a639fcdca 2024-03-26T04:41:20.741 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:20 smithi186 systemd[1]: Started Ceph mon.b for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:41:20.983 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:41:20.983 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:41:21.214 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 -- 172.21.15.186:0/2092172549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9200057d0 msgr2=0x7fd920007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/2092172549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9200057d0 0x7fd920007c60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd9341009b0 tx=0x7fd93000feb0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 -- 172.21.15.186:0/2092172549 shutdown_connections 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/2092172549 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd9200057d0 0x7fd920007c60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 -- 172.21.15.186:0/2092172549 >> 172.21.15.186:0/2092172549 conn(0x7fd9340fc590 msgr2=0x7fd9340fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 -- 172.21.15.186:0/2092172549 shutdown_connections 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.516+0000 7fd93c5e2700 1 -- 172.21.15.186:0/2092172549 wait complete. 2024-03-26T04:41:25.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93c5e2700 1 Processor -- start 2024-03-26T04:41:25.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93c5e2700 1 -- start start 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93c5e2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93c5e2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93419ab60 0x7fd934194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93c5e2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93419b080 con 0x7fd93419ab60 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93c5e2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd934194c80 con 0x7fd9200057d0 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.517+0000 7fd93a37e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93a37e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:38170/0 (socket says 172.21.15.186:38170) 2024-03-26T04:41:25.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93a37e700 1 -- 172.21.15.186:0/3142924576 learned_addr learned my addr 172.21.15.186:0/3142924576 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd939b7d700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93419ab60 0x7fd934194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93a37e700 1 -- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 msgr2=0x7fd93419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93a37e700 1 -- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 msgr2=0x7fd93419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93a37e700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93a37e700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd939b7d700 1 -- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 msgr2=0x7fd93419a620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd939b7d700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd939b7d700 1 -- 172.21.15.186:0/3142924576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93000f900 con 0x7fd93419ab60 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd939b7d700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93419ab60 0x7fd934194740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd92400b4d0 tx=0x7fd92400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd924011630 con 0x7fd93419ab60 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934194f10 con 0x7fd93419ab60 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd924011c10 con 0x7fd93419ab60 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.518+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd924010b80 con 0x7fd93419ab60 2024-03-26T04:41:25.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.519+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd934195460 con 0x7fd93419ab60 2024-03-26T04:41:25.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.519+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7fd92401a400 con 0x7fd93419ab60 2024-03-26T04:41:25.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.520+0000 7fd92b7fe700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fd918038d20 0x7fd91803b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:25.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.520+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd92404c510 con 0x7fd93419ab60 2024-03-26T04:41:25.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.520+0000 7fd93a37e700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fd918038d20 0x7fd91803b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:25.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.520+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93418e530 con 0x7fd93419ab60 2024-03-26T04:41:25.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.520+0000 7fd93a37e700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fd918038d20 0x7fd91803b1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd93000f8d0 tx=0x7fd9300057d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:25.525 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.525+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd924010e40 con 0x7fd93419ab60 2024-03-26T04:41:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: mon.a calling monitor election 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: mon.b calling monitor election 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: monmap e2: 2 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:41:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: fsmap 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: mgrmap e13: y(active, since 40s) 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: overall HEALTH_OK 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:25.811 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.811+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9340636b0 con 0x7fd93419ab60 2024-03-26T04:41:25.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:25.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: mon.a calling monitor election 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: mon.b calling monitor election 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: monmap e2: 2 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: fsmap 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: mgrmap e13: y(active, since 40s) 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: overall HEALTH_OK 2024-03-26T04:41:25.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:25.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:25.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:25.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:25.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:25.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.811+0000 7fd92b7fe700 1 -- 172.21.15.186:0/3142924576 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7fd9240172c0 con 0x7fd93419ab60 2024-03-26T04:41:25.813 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:41:25.813 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"c3b00422-eb2a-11ee-95d0-87774f69a715","modified":"2024-03-26T04:41:20.411452Z","created":"2024-03-26T04:39:41.201484Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:41:25.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fd918038d20 msgr2=0x7fd91803b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:25.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fd918038d20 0x7fd91803b1e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd93000f8d0 tx=0x7fd9300057d0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93419ab60 msgr2=0x7fd934194740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93419ab60 0x7fd934194740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd92400b4d0 tx=0x7fd92400b890 comp rx=0 tx=0).stop 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 shutdown_connections 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fd918038d20 0x7fd91803b1e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9200057d0 0x7fd93419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 --2- 172.21.15.186:0/3142924576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd93419ab60 0x7fd934194740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 >> 172.21.15.186:0/3142924576 conn(0x7fd9340fc590 msgr2=0x7fd9340fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 shutdown_connections 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:25.814+0000 7fd93c5e2700 1 -- 172.21.15.186:0/3142924576 wait complete. 2024-03-26T04:41:25.815 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-03-26T04:41:26.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:26 smithi186 ceph-mon[33438]: Deploying daemon mon.c on smithi012 2024-03-26T04:41:26.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/3142924576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:26.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:26 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:26.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:26 smithi012 ceph-mon[29863]: Deploying daemon mon.c on smithi012 2024-03-26T04:41:26.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/3142924576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:26.788 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:26 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:27.379 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:41:27.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:41:27.620 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:41:27.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:27 smithi012 ceph-mon[29863]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:27.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:27 smithi186 ceph-mon[33438]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:28.785 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:28 smithi012 ceph-mon[29863]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:28.944 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:28 smithi186 ceph-mon[33438]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:28 smithi012 systemd[1]: Started Ceph mon.c for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:41:34.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: mon.a calling monitor election 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: mon.b calling monitor election 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: mon.c calling monitor election 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: fsmap 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: mgrmap e13: y(active, since 49s) 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: overall HEALTH_OK 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: mon.a calling monitor election 2024-03-26T04:41:34.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: mon.b calling monitor election 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: mon.c calling monitor election 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: fsmap 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:34.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: mgrmap e13: y(active, since 49s) 2024-03-26T04:41:34.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: overall HEALTH_OK 2024-03-26T04:41:34.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: mon.a calling monitor election 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: mon.b calling monitor election 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: mon.c calling monitor election 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:34.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: monmap e3: 3 mons at {a=[v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: fsmap 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: mgrmap e13: y(active, since 49s) 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: overall HEALTH_OK 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:34.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 -- 172.21.15.186:0/499163452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cc0006240 msgr2=0x7f6cc00086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 --2- 172.21.15.186:0/499163452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cc0006240 0x7f6cc00086b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4102c30 tx=0x7f6cc400fe90 comp rx=0 tx=0).stop 2024-03-26T04:41:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 -- 172.21.15.186:0/499163452 shutdown_connections 2024-03-26T04:41:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 --2- 172.21.15.186:0/499163452 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cc0006240 0x7f6cc00086b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 -- 172.21.15.186:0/499163452 >> 172.21.15.186:0/499163452 conn(0x7f6cd40fd640 msgr2=0x7f6cd40ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 -- 172.21.15.186:0/499163452 shutdown_connections 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 -- 172.21.15.186:0/499163452 wait complete. 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.045+0000 7f6cdc525700 1 Processor -- start 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 -- start start 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd419ce10 0x7f6cd4196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 0x7f6cd41974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd419d4d0 con 0x7f6cd419ce10 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd41a37a0 con 0x7f6cc0006240 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdc525700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd41a38e0 con 0x7f6cd4197020 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34576/0 (socket says 172.21.15.186:34576) 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 -- 172.21.15.186:0/3575101711 learned_addr learned my addr 172.21.15.186:0/3575101711 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 msgr2=0x7f6cd419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 msgr2=0x7f6cd419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cd9ac0700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd419ce10 0x7f6cd4196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cda2c1700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdaac2700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 0x7f6cd41974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdaac2700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 msgr2=0x7f6cd41974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdaac2700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 msgr2=0x7f6cd41974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdaac2700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 0x7f6cd41974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cdaac2700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 0x7f6cd41974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cd9ac0700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 msgr2=0x7f6cd41974d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cd9ac0700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 0x7f6cd41974d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cd9ac0700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 msgr2=0x7f6cd419c8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cd9ac0700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.046+0000 7f6cd9ac0700 1 -- 172.21.15.186:0/3575101711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc400f8e0 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.047+0000 7f6cd9ac0700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd419ce10 0x7f6cd4196ae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000b550 tx=0x7f6cd000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.047+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0004230 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.047+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6cd0004390 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.047+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd41a3bc0 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.047+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00119d0 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.047+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd41a4110 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.048+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f6cd0011b30 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.048+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd41032b0 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.048+0000 7f6ccb7fe700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f6cb8039410 0x7f6cb803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.048+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6cd004b670 con 0x7f6cd419ce10 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.048+0000 7f6cda2c1700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f6cb8039410 0x7f6cb803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:35.052 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.048+0000 7f6cda2c1700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f6cb8039410 0x7f6cb803b8d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400f550 tx=0x7f6cc400f4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.053+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f6cd001a400 con 0x7f6cd419ce10 2024-03-26T04:41:35.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:35.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[29863]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:35.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:35.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:35.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:35.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[41141]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:35.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:35.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:35.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:35.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:35 smithi186 ceph-mon[33438]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:35.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:35.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:35.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.334+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cd4108eb0 con 0x7f6cd419ce10 2024-03-26T04:41:35.335 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.334+0000 7f6ccb7fe700 1 -- 172.21.15.186:0/3575101711 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1281 (secure 0 0 0) 0x7f6cd000c150 con 0x7f6cd419ce10 2024-03-26T04:41:35.335 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-26T04:41:35.336 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"c3b00422-eb2a-11ee-95d0-87774f69a715","modified":"2024-03-26T04:41:28.920896Z","created":"2024-03-26T04:39:41.201484Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3301","nonce":0},{"type":"v1","addr":"172.21.15.12:6790","nonce":0}]},"addr":"172.21.15.12:6790/0","public_addr":"172.21.15.12:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T04:41:35.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f6cb8039410 msgr2=0x7f6cb803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f6cb8039410 0x7f6cb803b8d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400f550 tx=0x7f6cc400f4a0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd419ce10 msgr2=0x7f6cd4196ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd419ce10 0x7f6cd4196ae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000b550 tx=0x7f6cd000b910 comp rx=0 tx=0).stop 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 shutdown_connections 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f6cb8039410 0x7f6cb803b8d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6cd4197020 0x7f6cd41974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0006240 0x7f6cd419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 --2- 172.21.15.186:0/3575101711 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6cd419ce10 0x7f6cd4196ae0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.336+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 >> 172.21.15.186:0/3575101711 conn(0x7f6cd40fd640 msgr2=0x7f6cd41057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.337+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 shutdown_connections 2024-03-26T04:41:35.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:35.337+0000 7f6cdc525700 1 -- 172.21.15.186:0/3575101711 wait complete. 2024-03-26T04:41:35.339 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-03-26T04:41:36.005 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T04:41:36.005 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T04:41:36.247 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:36.573 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.573 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.573 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/3575101711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.574 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/3575101711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.575 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:36.577 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:36.577 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:36.671 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/3575101711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.673 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.673 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:36.673 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:36.673 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:36.673 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:37.578 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[41141]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[41141]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[41141]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[41141]: Reconfiguring daemon mon.a on smithi012 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[41141]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[29863]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[29863]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[29863]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[29863]: Reconfiguring daemon mon.a on smithi012 2024-03-26T04:41:37.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:37 smithi012 ceph-mon[29863]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:37.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:37 smithi186 ceph-mon[33438]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:37.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:37 smithi186 ceph-mon[33438]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:41:37.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:37 smithi186 ceph-mon[33438]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:41:37.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:37 smithi186 ceph-mon[33438]: Reconfiguring daemon mon.a on smithi012 2024-03-26T04:41:37.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:37 smithi186 ceph-mon[33438]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 -- 172.21.15.12:0/1583047696 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06a820 msgr2=0x7f4f5c06ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/1583047696 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06a820 0x7f4f5c06ac00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4f50009990 tx=0x7f4f50009ca0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 -- 172.21.15.12:0/1583047696 shutdown_connections 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/1583047696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5c107400 0x7f4f5c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/1583047696 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06a820 0x7f4f5c06ac00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/1583047696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06b1d0 0x7f4f5c106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 -- 172.21.15.12:0/1583047696 >> 172.21.15.12:0/1583047696 conn(0x7f4f5c076fd0 msgr2=0x7f4f5c0773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 -- 172.21.15.12:0/1583047696 shutdown_connections 2024-03-26T04:41:38.682 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.681+0000 7f4f62c0b700 1 -- 172.21.15.12:0/1583047696 wait complete. 2024-03-26T04:41:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 Processor -- start 2024-03-26T04:41:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 -- start start 2024-03-26T04:41:38.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 0x7f4f5c19d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06b1d0 0x7f4f5c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5c107400 0x7f4f5c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c198000 con 0x7f4f5c06a820 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c198140 con 0x7f4f5c107400 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f62c0b700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c198300 con 0x7f4f5c06b1d0 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f609a7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 0x7f4f5c19d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f5bfff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06b1d0 0x7f4f5c19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f609a7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 0x7f4f5c19d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60896/0 (socket says 172.21.15.12:60896) 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f609a7700 1 -- 172.21.15.12:0/3945142951 learned_addr learned my addr 172.21.15.12:0/3945142951 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.682+0000 7f4f611a8700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5c107400 0x7f4f5c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f609a7700 1 -- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06b1d0 msgr2=0x7f4f5c19dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:38.685 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f609a7700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06b1d0 0x7f4f5c19dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f609a7700 1 -- 172.21.15.12:0/3945142951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5c107400 msgr2=0x7f4f5c1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f609a7700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5c107400 0x7f4f5c1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f609a7700 1 -- 172.21.15.12:0/3945142951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f48009520 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f5bfff700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06b1d0 0x7f4f5c19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f609a7700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 0x7f4f5c19d7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4f50004730 tx=0x7f4f50004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5001d070 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4f5000be20 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50005290 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f500095f0 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.683+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c100fb0 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.684+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f4f5000b9a0 con 0x7f4f5c06a820 2024-03-26T04:41:38.686 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.684+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c1037b0 con 0x7f4f5c06a820 2024-03-26T04:41:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.684+0000 7f4f59ffb700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f4c0391c0 0x7f4f4c03b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.685+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f5004cd20 con 0x7f4f5c06a820 2024-03-26T04:41:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.685+0000 7f4f5bfff700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f4c0391c0 0x7f4f4c03b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:38.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.685+0000 7f4f5bfff700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f4c0391c0 0x7f4f4c03b680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009fd0 tx=0x7f4f48012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:38.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.690+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4f50016720 con 0x7f4f5c06a820 2024-03-26T04:41:38.895 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.895+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4f5c198ed0 con 0x7f4f5c06a820 2024-03-26T04:41:38.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.895+0000 7f4f59ffb700 1 -- 172.21.15.12:0/3945142951 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f4f50016930 con 0x7f4f5c06a820 2024-03-26T04:41:38.896 INFO:teuthology.orchestra.run.smithi012.stdout:# minimal ceph.conf for c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:41:38.896 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-03-26T04:41:38.896 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:41:38.896 INFO:teuthology.orchestra.run.smithi012.stdout: mon_host = [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2024-03-26T04:41:38.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.897+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f4c0391c0 msgr2=0x7f4f4c03b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:38.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.897+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f4c0391c0 0x7f4f4c03b680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009fd0 tx=0x7f4f48012040 comp rx=0 tx=0).stop 2024-03-26T04:41:38.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.897+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 msgr2=0x7f4f5c19d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:38.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.897+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 0x7f4f5c19d7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4f50004730 tx=0x7f4f50004760 comp rx=0 tx=0).stop 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 shutdown_connections 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4f4c0391c0 0x7f4f4c03b680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4f5c06a820 0x7f4f5c19d7e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5c107400 0x7f4f5c1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 --2- 172.21.15.12:0/3945142951 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4f5c06b1d0 0x7f4f5c19dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 >> 172.21.15.12:0/3945142951 conn(0x7f4f5c076fd0 msgr2=0x7f4f5c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 shutdown_connections 2024-03-26T04:41:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:38.898+0000 7f4f62c0b700 1 -- 172.21.15.12:0/3945142951 wait complete. 2024-03-26T04:41:39.581 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: Reconfiguring daemon mon.c on smithi012 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3945142951' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:39.582 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: Reconfiguring daemon mon.c on smithi012 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:39.583 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3945142951' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:39.641 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T04:41:39.642 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:41:39.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:41:39.671 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:41:39.671 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: Reconfiguring daemon mon.c on smithi012 2024-03-26T04:41:39.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:39.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3945142951' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:39.738 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:41:39.738 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:41:39.773 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:41:39.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:41:39.843 INFO:tasks.cephadm:Adding mgr.y on smithi012 2024-03-26T04:41:39.843 INFO:tasks.cephadm:Adding mgr.x on smithi186 2024-03-26T04:41:39.843 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi012=y;smithi186=x' 2024-03-26T04:41:40.116 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:41:40.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[29863]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:40.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:40.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:40.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[41141]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:40.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:40.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:40 smithi186 ceph-mon[33438]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:40.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:40.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:40.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:41:40.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:41:40.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:41 smithi186 ceph-mon[33438]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:41:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:41 smithi186 ceph-mon[33438]: Reconfiguring daemon mon.b on smithi186 2024-03-26T04:41:42.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:41 smithi012 ceph-mon[29863]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:41:42.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:41 smithi012 ceph-mon[29863]: Reconfiguring daemon mon.b on smithi186 2024-03-26T04:41:42.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:41 smithi012 ceph-mon[41141]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:41:42.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:41 smithi012 ceph-mon[41141]: Reconfiguring daemon mon.b on smithi186 2024-03-26T04:41:42.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 -- 172.21.15.186:0/380782800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c102630 msgr2=0x7f1e0c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:42.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 --2- 172.21.15.186:0/380782800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c102630 0x7f1e0c104a20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00b0a0 tx=0x7f1dfc00b3b0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 -- 172.21.15.186:0/380782800 shutdown_connections 2024-03-26T04:41:42.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 --2- 172.21.15.186:0/380782800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c102630 0x7f1e0c104a20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 -- 172.21.15.186:0/380782800 >> 172.21.15.186:0/380782800 conn(0x7f1e0c0fc060 msgr2=0x7f1e0c0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:42.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 -- 172.21.15.186:0/380782800 shutdown_connections 2024-03-26T04:41:42.673 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.671+0000 7f1e144a0700 1 -- 172.21.15.186:0/380782800 wait complete. 2024-03-26T04:41:42.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 Processor -- start 2024-03-26T04:41:42.675 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 -- start start 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 0x7f1e0c196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e0c196c40 0x7f1e0c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c1970c0 0x7f1e0c19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c19e9e0 con 0x7f1e0c1970c0 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c19eb50 con 0x7f1e0c102630 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e144a0700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c19ed10 con 0x7f1e0c196c40 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e1223c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 0x7f1e0c196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e1223c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 0x7f1e0c196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34578/0 (socket says 172.21.15.186:34578) 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.672+0000 7f1e1223c700 1 -- 172.21.15.186:0/1602526101 learned_addr learned my addr 172.21.15.186:0/1602526101 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e11a3b700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e0c196c40 0x7f1e0c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e12a3d700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c1970c0 0x7f1e0c19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e1223c700 1 -- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e0c196c40 msgr2=0x7f1e0c19bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:42.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e1223c700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e0c196c40 0x7f1e0c19bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e1223c700 1 -- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c1970c0 msgr2=0x7f1e0c19e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e1223c700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c1970c0 0x7f1e0c19e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e1223c700 1 -- 172.21.15.186:0/1602526101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc009d00 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e11a3b700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e0c196c40 0x7f1e0c19bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e12a3d700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c1970c0 0x7f1e0c19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e1223c700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 0x7f1e0c196700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009850 tx=0x7f1dfc007e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc00e040 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1dfc00be70 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c19ef90 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0127d0 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.673+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c19f420 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.674+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f1dfc0050d0 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.674+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0c1903d0 con 0x7f1e0c102630 2024-03-26T04:41:42.677 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.674+0000 7f1e0b7fe700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1df4039410 0x7f1df403b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:42.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.674+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1dfc04c590 con 0x7f1e0c102630 2024-03-26T04:41:42.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.674+0000 7f1e11a3b700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1df4039410 0x7f1df403b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:42.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.675+0000 7f1e11a3b700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1df4039410 0x7f1df403b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e000079b0 tx=0x7f1e00008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:42.680 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.679+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1dfc0219b0 con 0x7f1e0c102630 2024-03-26T04:41:42.888 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.887+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi012=y;smithi186=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e0c02da80 con 0x7f1df4039410 2024-03-26T04:41:42.894 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.894+0000 7f1e0b7fe700 1 -- 172.21.15.186:0/1602526101 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1e0c02da80 con 0x7f1df4039410 2024-03-26T04:41:42.895 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1df4039410 msgr2=0x7f1df403b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1df4039410 0x7f1df403b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e000079b0 tx=0x7f1e00008040 comp rx=0 tx=0).stop 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 msgr2=0x7f1e0c196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 0x7f1e0c196700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009850 tx=0x7f1dfc007e90 comp rx=0 tx=0).stop 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 shutdown_connections 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1df4039410 0x7f1df403b8d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e0c1970c0 0x7f1e0c19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e0c102630 0x7f1e0c196700 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 --2- 172.21.15.186:0/1602526101 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e0c196c40 0x7f1e0c19bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 >> 172.21.15.186:0/1602526101 conn(0x7f1e0c0fc060 msgr2=0x7f1e0c0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:42.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 shutdown_connections 2024-03-26T04:41:42.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:41:42.896+0000 7f1e144a0700 1 -- 172.21.15.186:0/1602526101 wait complete. 2024-03-26T04:41:43.496 DEBUG:teuthology.orchestra.run.smithi186:mgr.x> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T04:41:43.498 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T04:41:43.498 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:41:43.498 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:41:43.518 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:41:43.518 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-03-26T04:41:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:41:43.576 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:43.576 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T04:41:43.576 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:43.576 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:43.577 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-26 04:41:28.400926209 +0000 2024-03-26T04:41:43.577 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-26 04:36:17.834638019 +0000 2024-03-26T04:41:43.577 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-26 04:36:17.834638019 +0000 2024-03-26T04:41:43.577 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-26 04:36:17.834638019 +0000 2024-03-26T04:41:43.577 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:41:43.646 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-26T04:41:43.647 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-26T04:41:43.647 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000278928 s, 1.8 MB/s 2024-03-26T04:41:43.648 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:41:43.706 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-26 04:41:28.401926201 +0000 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-26 04:36:18.184634829 +0000 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-26 04:36:18.184634829 +0000 2024-03-26T04:41:43.764 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-26 04:36:18.184634829 +0000 2024-03-26T04:41:43.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:43.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:43.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:43.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:41:43.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:43.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:43.831 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.832 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:43.833 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-26T04:41:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-26T04:41:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000239083 s, 2.1 MB/s 2024-03-26T04:41:43.837 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:41:43.894 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-03-26T04:41:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:41:43.949 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:43.949 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T04:41:43.949 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:43.950 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:43.950 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-26 04:41:28.401926201 +0000 2024-03-26T04:41:43.950 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-26 04:36:18.552631475 +0000 2024-03-26T04:41:43.950 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-26 04:36:18.552631475 +0000 2024-03-26T04:41:43.950 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-26 04:36:18.552631475 +0000 2024-03-26T04:41:43.950 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:41:44.014 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-26T04:41:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-26T04:41:44.015 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000208766 s, 2.5 MB/s 2024-03-26T04:41:44.016 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:41:44.071 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-03-26 04:41:28.402926192 +0000 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-03-26 04:36:18.891628385 +0000 2024-03-26T04:41:44.125 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-03-26 04:36:18.891628385 +0000 2024-03-26T04:41:44.126 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-03-26 04:36:18.891628385 +0000 2024-03-26T04:41:44.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:41:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-03-26T04:41:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-03-26T04:41:44.190 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000210168 s, 2.4 MB/s 2024-03-26T04:41:44.192 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:41:44.247 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:41:44.247 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:41:44.264 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:41:44.264 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-26 04:41:19.439215034 +0000 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-26 04:36:14.729330842 +0000 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-26 04:36:14.729330842 +0000 2024-03-26T04:41:44.320 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-26 04:36:14.729330842 +0000 2024-03-26T04:41:44.321 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:41:44.386 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-26T04:41:44.386 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-26T04:41:44.386 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000188323 s, 2.7 MB/s 2024-03-26T04:41:44.387 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:41:44.443 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-26 04:41:19.440215016 +0000 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-26 04:36:15.071324605 +0000 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-26 04:36:15.071324605 +0000 2024-03-26T04:41:44.499 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-26 04:36:15.071324605 +0000 2024-03-26T04:41:44.499 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:41:44.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:44 smithi186 ceph-mon[33438]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi012=y;smithi186=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:44.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:44 smithi186 ceph-mon[33438]: Saving service mgr spec with placement smithi012=y;smithi186=x;count:2 2024-03-26T04:41:44.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:44 smithi186 ceph-mon[33438]: Deploying daemon mgr.x on smithi186 2024-03-26T04:41:44.713 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-26T04:41:44.714 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-26T04:41:44.714 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000200734 s, 2.6 MB/s 2024-03-26T04:41:44.714 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:41:44.729 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-03-26T04:41:44.788 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:41:44.788 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:44.788 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T04:41:44.788 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:44.788 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:44.788 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-26 04:41:44.666761773 +0000 2024-03-26T04:41:44.789 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-26 04:36:15.414318351 +0000 2024-03-26T04:41:44.789 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-26 04:36:15.414318351 +0000 2024-03-26T04:41:44.789 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-26 04:36:15.414318351 +0000 2024-03-26T04:41:44.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:41:44.891 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-26T04:41:44.892 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-26T04:41:44.892 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000174926 s, 2.9 MB/s 2024-03-26T04:41:44.893 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:41:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:44 smithi012 ceph-mon[29863]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi012=y;smithi186=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:44 smithi012 ceph-mon[29863]: Saving service mgr spec with placement smithi012=y;smithi186=x;count:2 2024-03-26T04:41:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:44 smithi012 ceph-mon[29863]: Deploying daemon mgr.x on smithi186 2024-03-26T04:41:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:44 smithi012 ceph-mon[41141]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi012=y;smithi186=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:44 smithi012 ceph-mon[41141]: Saving service mgr spec with placement smithi012=y;smithi186=x;count:2 2024-03-26T04:41:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:44 smithi012 ceph-mon[41141]: Deploying daemon mgr.x on smithi186 2024-03-26T04:41:44.908 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-03-26T04:41:44.963 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:44 smithi186 systemd[1]: Starting Ceph mgr.x for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-26 04:41:44.848758502 +0000 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-26 04:36:15.770311859 +0000 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-26 04:36:15.770311859 +0000 2024-03-26T04:41:44.965 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-26 04:36:15.770311859 +0000 2024-03-26T04:41:44.966 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:41:45.032 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-26T04:41:45.033 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-26T04:41:45.033 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000176268 s, 2.9 MB/s 2024-03-26T04:41:45.034 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:41:45.090 INFO:tasks.cephadm:Deploying osd.0 on smithi012 with /dev/vg_nvme/lv_4... 2024-03-26T04:41:45.090 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:41:45.326 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:45.356 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 podman[35285]: 2024-03-26 04:41:45.039583083 +0000 UTC m=+0.018377678 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:41:45.356 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 podman[35285]: 2024-03-26 04:41:45.138078023 +0000 UTC m=+0.116872655 container create 06682ec9514dff436e13a6a7bfa83f396932823a04f469336fd5a375e6162b19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T04:41:45.356 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 podman[35285]: 2024-03-26 04:41:45.357227249 +0000 UTC m=+0.336021838 container init 06682ec9514dff436e13a6a7bfa83f396932823a04f469336fd5a375e6162b19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T04:41:45.697 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[29863]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[41141]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.698 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:45 smithi186 ceph-mon[33438]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:45.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:45.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 podman[35285]: 2024-03-26 04:41:45.359741981 +0000 UTC m=+0.338536567 container start 06682ec9514dff436e13a6a7bfa83f396932823a04f469336fd5a375e6162b19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:41:45.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 ceph-mgr[35328]: -- 172.21.15.186:0/2888694254 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ebc3a681a0 con 0x55ebc3a3f000 2024-03-26T04:41:45.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 bash[35285]: 06682ec9514dff436e13a6a7bfa83f396932823a04f469336fd5a375e6162b19 2024-03-26T04:41:45.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 systemd[1]: Started Ceph mgr.x for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:41:46.213 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:45 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:45.932+0000 7feea038c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:41:46.213 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:46 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:46.028+0000 7feea038c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:41:46.598 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:46.599 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:46.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:46.599 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:46.626 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:46.627 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:46.964 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:46 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:46.737+0000 7feea038c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:41:46.964 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:46 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:46.833+0000 7feea038c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:41:47.215 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:41:47.215 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:41:47.216 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-26T04:41:47.216 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-26T04:41:47.216 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113377 s, 925 MB/s 2024-03-26T04:41:47.216 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-26T04:41:47.459 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:46 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:46.990+0000 7feea038c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:41:47.459 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.094+0000 7feea038c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:41:47.460 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.184+0000 7feea038c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:41:47.460 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.458+0000 7feea038c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.571+0000 7feea038c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.659+0000 7feea038c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:47.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:47.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:47.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:47.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:47.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:47.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:47.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:47.968 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_4 2024-03-26T04:41:48.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.755+0000 7feea038c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:41:48.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:47.938+0000 7feea038c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:41:48.903 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:48 smithi186 ceph-mon[33438]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:41:48.903 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:48 smithi186 ceph-mon[33438]: Reconfiguring daemon mgr.y on smithi012 2024-03-26T04:41:48.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:48 smithi012 ceph-mon[41141]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:41:48.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:48 smithi012 ceph-mon[41141]: Reconfiguring daemon mgr.y on smithi012 2024-03-26T04:41:48.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:48 smithi012 ceph-mon[29863]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:41:48.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:48 smithi012 ceph-mon[29863]: Reconfiguring daemon mgr.y on smithi012 2024-03-26T04:41:49.025 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:41:49.213 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:48 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:48.902+0000 7feea038c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:41:49.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:49.031+0000 7feea038c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:41:49.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-mon[33438]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:49.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:49.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:49.651 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:49.651 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:49.365+0000 7feea038c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:41:49.651 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:49.454+0000 7feea038c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:41:49.651 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:49.649+0000 7feea038c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:49.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[29863]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:49.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:49.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:49.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:49.862 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[41141]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:49.863 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:49.863 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:49.863 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:49 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:49.963 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:49 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:49.831+0000 7feea038c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:41:50.376 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:50.112+0000 7feea038c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:50.377 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:50.375+0000 7feea038c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:50.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 -- 172.21.15.12:0/2058322555 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 msgr2=0x7f4d18109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/2058322555 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d18109350 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009f90 tx=0x7f4d1000b330 comp rx=0 tx=0).stop 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 -- 172.21.15.12:0/2058322555 shutdown_connections 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/2058322555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18100e80 0x7f4d181012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/2058322555 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d18109350 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/2058322555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d18102070 0x7f4d181024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 -- 172.21.15.12:0/2058322555 >> 172.21.15.12:0/2058322555 conn(0x7f4d180fc450 msgr2=0x7f4d180fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 -- 172.21.15.12:0/2058322555 shutdown_connections 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.621+0000 7f4d1cba1700 1 -- 172.21.15.12:0/2058322555 wait complete. 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 Processor -- start 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 -- start start 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d18100e80 0x7f4d18197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:50.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 0x7f4d18197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:50.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d1819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:50.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1819eec0 con 0x7f4d18102070 2024-03-26T04:41:50.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1819f000 con 0x7f4d18100e80 2024-03-26T04:41:50.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d1cba1700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1819f190 con 0x7f4d18102a30 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d16ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 0x7f4d18197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d16ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 0x7f4d18197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56366/0 (socket says 172.21.15.12:56366) 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d17fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d1819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d16ffd700 1 -- 172.21.15.12:0/1805751058 learned_addr learned my addr 172.21.15.12:0/1805751058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.622+0000 7f4d177fe700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d18100e80 0x7f4d18197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d17fff700 1 -- 172.21.15.12:0/1805751058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d18100e80 msgr2=0x7f4d18197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d17fff700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d18100e80 0x7f4d18197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d17fff700 1 -- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 msgr2=0x7f4d18197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d17fff700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 0x7f4d18197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:41:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d17fff700 1 -- 172.21.15.12:0/1805751058 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d08009520 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d16ffd700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 0x7f4d18197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d17fff700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d1819e8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000d010 tx=0x7f4d10007d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000e040 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4d10004c40 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d10009c40 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100052a0 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.623+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d18103650 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.624+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 13) v1 ==== 45984+0+0 (secure 0 0 0) 0x7f4d10005400 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.624+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d18108ce0 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.624+0000 7f4d14ff9700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4d000391c0 0x7f4d0003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.624+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d10050e10 con 0x7f4d18102a30 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.624+0000 7f4d177fe700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4d000391c0 0x7f4d0003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:41:50.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.625+0000 7f4d177fe700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4d000391c0 0x7f4d0003b680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d18101be0 tx=0x7f4d08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:41:50.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.629+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4d100056b0 con 0x7f4d18102a30 2024-03-26T04:41:50.657 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:50.608+0000 7feea038c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:41:50.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:50.847+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d180624c0 con 0x7f4d000391c0 2024-03-26T04:41:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:50.848 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[29863]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:50.848 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:50.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:50.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:50.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:50.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:50 smithi012 ceph-mon[41141]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:50.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:50.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:50.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:50.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:41:50.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:50 smithi186 ceph-mon[33438]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:51.656 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:51.401+0000 7feea038c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:41:51.844 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[41141]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:51.844 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:41:51.844 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:41:51.844 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:51.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[29863]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:51.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:41:51.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:41:51.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:51.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:51 smithi186 ceph-mon[33438]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:51.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:41:51.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:41:51.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:52.274 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:51.966+0000 7feea038c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:41:52.274 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:52.063+0000 7feea038c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:41:52.274 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:41:52.274+0000 7feea038c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:41:52.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:41:52.664+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mgrmap(e 14) v1 ==== 91342+0+0 (secure 0 0 0) 0x7f4d10021d20 con 0x7f4d18102a30 2024-03-26T04:41:52.875 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[41141]: Standby manager daemon x started 2024-03-26T04:41:52.875 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:52.875 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:52.875 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:52.875 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:52.875 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[41141]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:52.876 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[29863]: Standby manager daemon x started 2024-03-26T04:41:52.876 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:52.876 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:52.876 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:52.876 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:52.876 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:52 smithi012 ceph-mon[29863]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:52.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-mon[33438]: Standby manager daemon x started 2024-03-26T04:41:52.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:52.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:52.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:52.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/2539466673' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:52.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:52 smithi186 ceph-mon[33438]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:53.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-26T04:41:53.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:41:53.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2471319012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]: dispatch 2024-03-26T04:41:53.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]: dispatch 2024-03-26T04:41:53.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]': finished 2024-03-26T04:41:53.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:41:53.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:53 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2471319012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]: dispatch 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]: dispatch 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]': finished 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:41:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: mgrmap e14: y(active, since 68s), standbys: x 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2471319012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]: dispatch 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]: dispatch 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ff69b85-c494-4958-862f-62bc607e0807"}]': finished 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:41:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:53 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:54.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3940213903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:41:54.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:54 smithi186 ceph-mon[33438]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:55.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3940213903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:41:55.121 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:54 smithi012 ceph-mon[29863]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:55.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3940213903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:41:55.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:54 smithi012 ceph-mon[41141]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:56.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:56 smithi186 ceph-mon[33438]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:56.972 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:56 smithi012 ceph-mon[29863]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:56.972 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:56 smithi012 ceph-mon[41141]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:58 smithi186 ceph-mon[33438]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:59.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:58 smithi012 ceph-mon[29863]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:41:59.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:58 smithi012 ceph-mon[41141]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:00.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:59 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:42:00.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:41:59 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:00.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:59 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:42:00.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:41:59 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:00.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:59 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:42:00.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:41:59 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:01.049 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:00 smithi012 ceph-mon[29863]: Deploying daemon osd.0 on smithi012 2024-03-26T04:42:01.049 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:00 smithi012 ceph-mon[29863]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:01.049 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:00 smithi012 ceph-mon[41141]: Deploying daemon osd.0 on smithi012 2024-03-26T04:42:01.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:00 smithi012 ceph-mon[41141]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:00 smithi186 ceph-mon[33438]: Deploying daemon osd.0 on smithi012 2024-03-26T04:42:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:00 smithi186 ceph-mon[33438]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:02 smithi012 ceph-mon[29863]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:02.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:02 smithi012 ceph-mon[41141]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:02.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:02 smithi186 ceph-mon[33438]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:04.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:04 smithi186 ceph-mon[33438]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:05.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:04 smithi012 ceph-mon[41141]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:05.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:04 smithi012 ceph-mon[29863]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:06.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:06.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:06.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:06.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:06.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:06.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:06.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:06.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:06.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:07.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:06 smithi012 ceph-mon[29863]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:07.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:06 smithi012 ceph-mon[41141]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:07.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:06 smithi186 ceph-mon[33438]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:08.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.597+0000 7f4d14ff9700 1 -- 172.21.15.12:0/1805751058 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4d180624c0 con 0x7f4d000391c0 2024-03-26T04:42:08.598 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 0 on host 'smithi012' 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4d000391c0 msgr2=0x7f4d0003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4d000391c0 0x7f4d0003b680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d18101be0 tx=0x7f4d08009260 comp rx=0 tx=0).stop 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 msgr2=0x7f4d1819e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d1819e8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000d010 tx=0x7f4d10007d00 comp rx=0 tx=0).stop 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 shutdown_connections 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f4d000391c0 0x7f4d0003b680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4d18102a30 0x7f4d1819e8f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d18100e80 0x7f4d18197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:08.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 --2- 172.21.15.12:0/1805751058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4d18102070 0x7f4d18197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:08.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.600+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 >> 172.21.15.12:0/1805751058 conn(0x7f4d180fc450 msgr2=0x7f4d180fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:42:08.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.601+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 shutdown_connections 2024-03-26T04:42:08.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:08.601+0000 7f4d1cba1700 1 -- 172.21.15.12:0/1805751058 wait complete. 2024-03-26T04:42:08.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[41141]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:08.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:08.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:08 smithi012 ceph-mon[29863]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:09.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:09.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:09.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:08 smithi186 ceph-mon[33438]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:09.265 DEBUG:teuthology.orchestra.run.smithi012:osd.0> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T04:42:09.267 INFO:tasks.cephadm:Deploying osd.1 on smithi012 with /dev/vg_nvme/lv_3... 2024-03-26T04:42:09.267 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:42:09.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[29863]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:42:09.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[29863]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:42:09.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:09.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[29863]: from='osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:09.868 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[29863]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:09.868 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[41141]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:42:09.868 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[41141]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:42:09.868 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:09.868 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[41141]: from='osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:09.868 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-mon[41141]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:09.868 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0[45181]: 2024-03-26T04:42:09.748+0000 7fb6826b0700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T04:42:09.868 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:42:09 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0[45181]: 2024-03-26T04:42:09.750+0000 7fb67e4a3700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:42:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:09 smithi186 ceph-mon[33438]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:42:10.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:09 smithi186 ceph-mon[33438]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:42:10.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:10.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:09 smithi186 ceph-mon[33438]: from='osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:10.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:09 smithi186 ceph-mon[33438]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:10.467 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:10.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[29863]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:10.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[29863]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:42:10.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:10.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:10.985 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[29863]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:10.986 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[41141]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:10.986 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[41141]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:42:10.986 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:10.986 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:10.986 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:10 smithi012 ceph-mon[41141]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:11.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:10 smithi186 ceph-mon[33438]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:11.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:10 smithi186 ceph-mon[33438]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:42:11.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:10 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:11.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:10 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:11.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:10 smithi186 ceph-mon[33438]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:42:11.890 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:42:11.890 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:42:11.890 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[29863]: osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] boot 2024-03-26T04:42:11.890 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[29863]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:42:11.890 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:11.891 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:42:11.891 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:42:11.891 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[41141]: osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] boot 2024-03-26T04:42:11.891 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[41141]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:42:11.891 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:11 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:12.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:11 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:42:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:11 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:42:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:11 smithi186 ceph-mon[33438]: osd.0 [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] boot 2024-03-26T04:42:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:11 smithi186 ceph-mon[33438]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:42:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:11 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:12.843 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:42:12.843 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:42:12.843 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-26T04:42:12.844 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-26T04:42:12.844 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123155 s, 851 MB/s 2024-03-26T04:42:12.844 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-26T04:42:13.142 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:12 smithi012 ceph-mon[29863]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:42:13.142 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:12 smithi012 ceph-mon[29863]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:13.142 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:12 smithi012 ceph-mon[41141]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:42:13.142 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:12 smithi012 ceph-mon[41141]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:13.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:12 smithi186 ceph-mon[33438]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:42:13.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:12 smithi186 ceph-mon[33438]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:13.427 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_3 2024-03-26T04:42:13.667 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:14.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi012 2024-03-26T04:42:14.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi012 to 16069M 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi012 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.088 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:14.089 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi012 to 16069M 2024-03-26T04:42:14.089 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.089 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:14.089 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:14.089 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi012 2024-03-26T04:42:14.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi012 to 16069M 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:14.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:14 smithi012 ceph-mon[29863]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:14.943 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:14 smithi012 ceph-mon[41141]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 -- 172.21.15.12:0/1998351021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e81022d0 msgr2=0x7f21e8108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 --2- 172.21.15.12:0/1998351021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e81022d0 0x7f21e8108b80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f21e0009fc0 tx=0x7f21e000b330 comp rx=0 tx=0).stop 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 -- 172.21.15.12:0/1998351021 shutdown_connections 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 --2- 172.21.15.12:0/1998351021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e81022d0 0x7f21e8108b80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 --2- 172.21.15.12:0/1998351021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21e8100730 0x7f21e8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 --2- 172.21.15.12:0/1998351021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e8101930 0x7f21e8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:15.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 -- 172.21.15.12:0/1998351021 >> 172.21.15.12:0/1998351021 conn(0x7f21e80fbcf0 msgr2=0x7f21e80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 -- 172.21.15.12:0/1998351021 shutdown_connections 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 -- 172.21.15.12:0/1998351021 wait complete. 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 Processor -- start 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.182+0000 7f21efcca700 1 -- start start 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21efcca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21e8100730 0x7f21e81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21efcca700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 0x7f21e8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21efcca700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 0x7f21e819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21efcca700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e819e050 con 0x7f21e8101930 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21efcca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e819e190 con 0x7f21e8100730 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21efcca700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e81094d0 con 0x7f21e81022d0 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 0x7f21e819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ed265700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 0x7f21e8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:15.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 0x7f21e819da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:39796/0 (socket says 172.21.15.12:39796) 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ed265700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 0x7f21e8196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36212/0 (socket says 172.21.15.12:36212) 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 -- 172.21.15.12:0/1653176007 learned_addr learned my addr 172.21.15.12:0/1653176007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21eda66700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21e8100730 0x7f21e81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 -- 172.21.15.12:0/1653176007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21e8100730 msgr2=0x7f21e81964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21e8100730 0x7f21e81964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 -- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 msgr2=0x7f21e8196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 0x7f21e8196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 -- 172.21.15.12:0/1653176007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e40095f0 con 0x7f21e81022d0 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ed265700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 0x7f21e8196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21ee267700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 0x7f21e819da80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007c80 tx=0x7f21e0007cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.183+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000e040 con 0x7f21e81022d0 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.184+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f21e000be00 con 0x7f21e81022d0 2024-03-26T04:42:15.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.184+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e0009c40 con 0x7f21e81022d0 2024-03-26T04:42:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.184+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e00052a0 con 0x7f21e81022d0 2024-03-26T04:42:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.184+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e81099e0 con 0x7f21e81022d0 2024-03-26T04:42:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.185+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e810c750 con 0x7f21e81022d0 2024-03-26T04:42:15.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.185+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 14) v1 ==== 91342+0+0 (secure 0 0 0) 0x7f21e001b450 con 0x7f21e81022d0 2024-03-26T04:42:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.186+0000 7f21deffd700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f21d406d940 0x7f21d406fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.186+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f21e008d6e0 con 0x7f21e81022d0 2024-03-26T04:42:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.186+0000 7f21eda66700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f21d406d940 0x7f21d406fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:15.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.186+0000 7f21eda66700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f21d406d940 0x7f21d406fe00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f21d8005fd0 tx=0x7f21d8005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:42:15.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.191+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f21e0058240 con 0x7f21e81022d0 2024-03-26T04:42:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:14 smithi186 ceph-mon[33438]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:15.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:15.409+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f21e810c0c0 con 0x7f21d406d940 2024-03-26T04:42:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[29863]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:42:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:42:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:16.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[41141]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:42:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:42:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:15 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:15 smithi186 ceph-mon[33438]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:15 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:42:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:15 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:42:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:15 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:17.135 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:16 smithi012 ceph-mon[29863]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:17.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:16 smithi012 ceph-mon[41141]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:17.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:16 smithi186 ceph-mon[33438]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:18.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/981281493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]: dispatch 2024-03-26T04:42:18.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]: dispatch 2024-03-26T04:42:18.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]': finished 2024-03-26T04:42:18.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[29863]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:42:18.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:18.643 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/981281493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]: dispatch 2024-03-26T04:42:18.644 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]: dispatch 2024-03-26T04:42:18.644 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]': finished 2024-03-26T04:42:18.644 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[41141]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:42:18.644 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:18 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/981281493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]: dispatch 2024-03-26T04:42:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]: dispatch 2024-03-26T04:42:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0dc7629c-67f1-498c-a7c8-434c9f9695c1"}]': finished 2024-03-26T04:42:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:18 smithi186 ceph-mon[33438]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:42:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:18 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:19 smithi012 ceph-mon[29863]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1141695769' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:42:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:19 smithi012 ceph-mon[41141]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1141695769' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:42:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:19 smithi186 ceph-mon[33438]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1141695769' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:42:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:20 smithi186 ceph-mon[33438]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:21.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:20 smithi012 ceph-mon[29863]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:21.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:20 smithi012 ceph-mon[41141]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:22.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:22 smithi012 ceph-mon[29863]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:22.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:22 smithi012 ceph-mon[41141]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:22.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:22 smithi186 ceph-mon[33438]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:24 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:42:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:24 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:24.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:24 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:42:24.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:24 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:24 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:42:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:24 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:25.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:25 smithi012 ceph-mon[29863]: Deploying daemon osd.1 on smithi012 2024-03-26T04:42:25.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:25 smithi012 ceph-mon[29863]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:25.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:25 smithi012 ceph-mon[41141]: Deploying daemon osd.1 on smithi012 2024-03-26T04:42:25.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:25 smithi012 ceph-mon[41141]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:25.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:25 smithi186 ceph-mon[33438]: Deploying daemon osd.1 on smithi012 2024-03-26T04:42:25.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:25 smithi186 ceph-mon[33438]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:26 smithi186 ceph-mon[33438]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:27.076 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:26 smithi012 ceph-mon[29863]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:27.076 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:26 smithi012 ceph-mon[41141]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:28.758 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:28 smithi012 ceph-mon[29863]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:28.759 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:28 smithi012 ceph-mon[41141]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:28 smithi186 ceph-mon[33438]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:30.813 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:30.813 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:30.813 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:30.813 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[29863]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:30.813 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:30.814 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:30.814 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:30.814 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:30 smithi012 ceph-mon[41141]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:30.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:30 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:30.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:30 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:30.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:30 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:30.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:30 smithi186 ceph-mon[33438]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:33.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.212+0000 7f21deffd700 1 -- 172.21.15.12:0/1653176007 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f21e810c0c0 con 0x7f21d406d940 2024-03-26T04:42:33.213 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 1 on host 'smithi012' 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.214+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f21d406d940 msgr2=0x7f21d406fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.214+0000 7f21efcca700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f21d406d940 0x7f21d406fe00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f21d8005fd0 tx=0x7f21d8005e50 comp rx=0 tx=0).stop 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.214+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 msgr2=0x7f21e819da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.214+0000 7f21efcca700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 0x7f21e819da80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007c80 tx=0x7f21e0007cb0 comp rx=0 tx=0).stop 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 shutdown_connections 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f21d406d940 0x7f21d406fe00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:33.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f21e81022d0 0x7f21e819da80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21e8100730 0x7f21e81964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 --2- 172.21.15.12:0/1653176007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f21e8101930 0x7f21e8196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 >> 172.21.15.12:0/1653176007 conn(0x7f21e80fbcf0 msgr2=0x7f21e8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:42:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 shutdown_connections 2024-03-26T04:42:33.216 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:33.215+0000 7f21efcca700 1 -- 172.21.15.12:0/1653176007 wait complete. 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.542 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:33 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:33.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:33 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:33.868 DEBUG:teuthology.orchestra.run.smithi012:osd.1> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T04:42:33.871 INFO:tasks.cephadm:Deploying osd.2 on smithi012 with /dev/vg_nvme/lv_2... 2024-03-26T04:42:33.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:42:34.569 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:34.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[29863]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[29863]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[29863]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[41141]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[41141]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[41141]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:34.839 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:34.840 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T04:42:34.494+0000 7fbe2f29f700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T04:42:34.840 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:42:34 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T04:42:34.497+0000 7fbe2b092700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:42:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:34 smithi186 ceph-mon[33438]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:42:34.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:34 smithi186 ceph-mon[33438]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:42:34.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:34 smithi186 ceph-mon[33438]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:34.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:34 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[29863]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:35.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[29863]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:42:35.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[29863]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:35.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[41141]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:35.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[41141]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:42:35.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:35 smithi012 ceph-mon[41141]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:35 smithi186 ceph-mon[33438]: from='osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:35.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:35 smithi186 ceph-mon[33438]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:42:35.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:35.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:35 smithi186 ceph-mon[33438]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:42:36.717 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:42:36.717 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:42:36.717 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[29863]: osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] boot 2024-03-26T04:42:36.717 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[29863]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:42:36.717 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:36.718 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:42:36.718 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:42:36.718 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[41141]: osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] boot 2024-03-26T04:42:36.718 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[41141]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:42:36.718 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:36.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:36 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:42:36.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:36 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:42:36.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:36 smithi186 ceph-mon[33438]: osd.1 [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] boot 2024-03-26T04:42:36.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:36 smithi186 ceph-mon[33438]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:42:36.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:37.658 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:42:37.658 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:42:37.658 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-26T04:42:37.658 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-26T04:42:37.658 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150841 s, 695 MB/s 2024-03-26T04:42:37.658 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-26T04:42:37.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:37 smithi012 ceph-mon[29863]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:42:37.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:37 smithi012 ceph-mon[29863]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:37.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:37 smithi012 ceph-mon[41141]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:42:37.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:37 smithi012 ceph-mon[41141]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:37.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:37 smithi186 ceph-mon[33438]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:42:37.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:37 smithi186 ceph-mon[33438]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:38.248 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_2 2024-03-26T04:42:38.490 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:38.848 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi012 2024-03-26T04:42:38.848 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.848 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.848 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:38.848 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi012 to 8034M 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi012 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:38.849 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi012 to 8034M 2024-03-26T04:42:38.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:38.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:38.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi012 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi012 to 8034M 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:39.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:39 smithi012 ceph-mon[29863]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:39.840 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:39 smithi012 ceph-mon[41141]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:39.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:39 smithi186 ceph-mon[33438]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:40.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 -- 172.21.15.12:0/1388601824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2408100130 msgr2=0x7f24081005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:40.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 --2- 172.21.15.12:0/1388601824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2408100130 0x7f24081005b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009990 tx=0x7f23fc009ca0 comp rx=0 tx=0).stop 2024-03-26T04:42:40.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 -- 172.21.15.12:0/1388601824 shutdown_connections 2024-03-26T04:42:40.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 --2- 172.21.15.12:0/1388601824 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100af0 0x7f240810a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:40.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 --2- 172.21.15.12:0/1388601824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24081036b0 0x7f2408103ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 --2- 172.21.15.12:0/1388601824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2408100130 0x7f24081005b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 -- 172.21.15.12:0/1388601824 >> 172.21.15.12:0/1388601824 conn(0x7f24080fbcf0 msgr2=0x7f24080fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 -- 172.21.15.12:0/1388601824 shutdown_connections 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.053+0000 7f2410779700 1 -- 172.21.15.12:0/1388601824 wait complete. 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 Processor -- start 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 -- start start 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 0x7f2408102db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2408100af0 0x7f2408101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24081036b0 0x7f2408101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24081020c0 con 0x7f24081036b0 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2408102230 con 0x7f2408100af0 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f2410779700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24081a1630 con 0x7f2408100130 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 0x7f2408102db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:40.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 0x7f2408102db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50736/0 (socket says 172.21.15.12:50736) 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 -- 172.21.15.12:0/2341828539 learned_addr learned my addr 172.21.15.12:0/2341828539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240ed16700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24081036b0 0x7f2408101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240dd14700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2408100af0 0x7f2408101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 -- 172.21.15.12:0/2341828539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2408100af0 msgr2=0x7f2408101400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2408100af0 0x7f2408101400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 -- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24081036b0 msgr2=0x7f2408101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24081036b0 0x7f2408101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 -- 172.21.15.12:0/2341828539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009520 con 0x7f2408100130 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.054+0000 7f240e515700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 0x7f2408102db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f240000ada0 tx=0x7f240000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.055+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240000f070 con 0x7f2408100130 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.055+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f240000ae80 con 0x7f2408100130 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.055+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc0095f0 con 0x7f2408100130 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.055+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400004280 con 0x7f2408100130 2024-03-26T04:42:40.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.055+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24081a1c80 con 0x7f2408100130 2024-03-26T04:42:40.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.056+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2408109690 con 0x7f2408100130 2024-03-26T04:42:40.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.056+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 14) v1 ==== 91342+0+0 (secure 0 0 0) 0x7f2400003a20 con 0x7f2408100130 2024-03-26T04:42:40.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.057+0000 7f24077fe700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f23f006d940 0x7f23f006fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:42:40.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.057+0000 7f240dd14700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f23f006d940 0x7f23f006fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:42:40.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.057+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2199+0+0 (secure 0 0 0) 0x7f240008b880 con 0x7f2408100130 2024-03-26T04:42:40.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.057+0000 7f240dd14700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f23f006d940 0x7f23f006fe00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0058e0 tx=0x7f23fc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:42:40.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.062+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2400056230 con 0x7f2408100130 2024-03-26T04:42:40.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:40.273+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f240810d1a0 con 0x7f23f006d940 2024-03-26T04:42:40.873 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:42:40.873 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:42:40.873 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:40.873 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:42:40.873 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:42:40.873 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:40.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:42:40.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:42:40.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:41.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:41 smithi012 ceph-mon[29863]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:41.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:41 smithi012 ceph-mon[29863]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:41.745 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:41 smithi012 ceph-mon[41141]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:41.745 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:41 smithi012 ceph-mon[41141]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:41.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:41 smithi186 ceph-mon[33438]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:41.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:41 smithi186 ceph-mon[33438]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:42 smithi186 ceph-mon[33438]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:43.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:42 smithi012 ceph-mon[29863]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:43.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:42 smithi012 ceph-mon[41141]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4216998346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185f5e96-6db7-41a5-9384-384dae329880"}]: dispatch 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4216998346' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "185f5e96-6db7-41a5-9384-384dae329880"}]': finished 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[29863]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1417839197' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4216998346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185f5e96-6db7-41a5-9384-384dae329880"}]: dispatch 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4216998346' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "185f5e96-6db7-41a5-9384-384dae329880"}]': finished 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[41141]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1417839197' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:42:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4216998346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "185f5e96-6db7-41a5-9384-384dae329880"}]: dispatch 2024-03-26T04:42:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4216998346' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "185f5e96-6db7-41a5-9384-384dae329880"}]': finished 2024-03-26T04:42:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:43 smithi186 ceph-mon[33438]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:42:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1417839197' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:42:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:44 smithi186 ceph-mon[33438]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:45.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:44 smithi012 ceph-mon[29863]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:45.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:44 smithi012 ceph-mon[41141]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:46.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:46 smithi012 ceph-mon[29863]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:46.939 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:46 smithi012 ceph-mon[41141]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:46 smithi186 ceph-mon[33438]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:48.865 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:48 smithi012 ceph-mon[29863]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:48.866 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:48 smithi012 ceph-mon[41141]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:48.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:48 smithi186 ceph-mon[33438]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:49.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:49 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:42:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:49 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:49 smithi186 ceph-mon[33438]: Deploying daemon osd.2 on smithi012 2024-03-26T04:42:49.977 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:49 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:42:49.977 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:49 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:49.977 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:49 smithi012 ceph-mon[41141]: Deploying daemon osd.2 on smithi012 2024-03-26T04:42:49.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:42:49.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:49.978 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:49 smithi012 ceph-mon[29863]: Deploying daemon osd.2 on smithi012 2024-03-26T04:42:50.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:50 smithi186 ceph-mon[33438]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:51.015 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:50 smithi012 ceph-mon[29863]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:51.016 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:50 smithi012 ceph-mon[41141]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:52.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:52 smithi012 ceph-mon[29863]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:52.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:52 smithi012 ceph-mon[41141]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:52.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:52 smithi186 ceph-mon[33438]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:55.821 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[29863]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:55.822 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:55 smithi012 ceph-mon[41141]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:55.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:55 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:55.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:55 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:55.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:55 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:55.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:55 smithi186 ceph-mon[33438]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:56.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:56 smithi012 ceph-mon[29863]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:56.895 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:56 smithi012 ceph-mon[41141]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:56.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:56 smithi186 ceph-mon[33438]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:57.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:57 smithi012 ceph-mon[29863]: from='osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:42:57.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:57 smithi012 ceph-mon[29863]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:42:57.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:57 smithi012 ceph-mon[41141]: from='osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:42:57.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:57 smithi012 ceph-mon[41141]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:42:57.953 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 2 on host 'smithi012' 2024-03-26T04:42:57.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.952+0000 7f24077fe700 1 -- 172.21.15.12:0/2341828539 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f240810d1a0 con 0x7f23f006d940 2024-03-26T04:42:57.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f23f006d940 msgr2=0x7f23f006fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:57.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f23f006d940 0x7f23f006fe00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0058e0 tx=0x7f23fc00b260 comp rx=0 tx=0).stop 2024-03-26T04:42:57.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 msgr2=0x7f2408102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:42:57.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 0x7f2408102db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f240000ada0 tx=0x7f240000c330 comp rx=0 tx=0).stop 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 shutdown_connections 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f23f006d940 0x7f23f006fe00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f24081036b0 0x7f2408101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2408100130 0x7f2408102db0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 --2- 172.21.15.12:0/2341828539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2408100af0 0x7f2408101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 >> 172.21.15.12:0/2341828539 conn(0x7f24080fbcf0 msgr2=0x7f24081067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 shutdown_connections 2024-03-26T04:42:57.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:42:57.955+0000 7f2410779700 1 -- 172.21.15.12:0/2341828539 wait complete. 2024-03-26T04:42:57.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:57 smithi186 ceph-mon[33438]: from='osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:42:57.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:57 smithi186 ceph-mon[33438]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:42:58.571 DEBUG:teuthology.orchestra.run.smithi012:osd.2> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:42:58.573 INFO:tasks.cephadm:Deploying osd.3 on smithi012 with /dev/vg_nvme/lv_1... 2024-03-26T04:42:58.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:42:58.865 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:58.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.907 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2[57219]: 2024-03-26T04:42:58.697+0000 7fa9d8528700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T04:42:58.907 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:42:58 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2[57219]: 2024-03-26T04:42:58.700+0000 7fa9d431b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:42:58.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:58.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:58 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:42:59.854 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[29863]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:59.854 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[29863]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:42:59.854 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:59.854 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:59.855 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[41141]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:59.855 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[41141]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:42:59.855 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:59.855 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:42:59 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:59.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:59 smithi186 ceph-mon[33438]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:42:59.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:59 smithi186 ceph-mon[33438]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:42:59.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:59 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:59.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:42:59 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:43:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:43:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:43:00.740 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-03-26T04:43:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-03-26T04:43:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0153889 s, 681 MB/s 2024-03-26T04:43:00.740 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-03-26T04:43:01.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:43:01.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:43:01.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[29863]: osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] boot 2024-03-26T04:43:01.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[29863]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[29863]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[41141]: osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] boot 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[41141]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:43:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:00 smithi012 ceph-mon[41141]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:00 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:43:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:00 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:43:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:00 smithi186 ceph-mon[33438]: osd.2 [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] boot 2024-03-26T04:43:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:00 smithi186 ceph-mon[33438]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:43:01.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:00 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:43:01.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:00 smithi186 ceph-mon[33438]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:02.108 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[29863]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[41141]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:43:02.109 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:01 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:43:02.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:01 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:43:02.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:01 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:43:02.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:01 smithi186 ceph-mon[33438]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:43:02.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:01 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:43:02.360 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi012:vg_nvme/lv_1 2024-03-26T04:43:02.630 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[60852]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:43:02.630 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[60852]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:02.630 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[60852]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:02.630 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[60852]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:02.875 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:43:02.905 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[61362]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:43:02.905 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[61362]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:02.905 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[61362]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:02.905 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 04:43:02 smithi012 sudo[61362]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:03.287 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:43:03.287 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:03.287 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:03.287 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61386]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:03.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:43:03.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[29863]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:43:03.288 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[29863]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:03.288 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:03.288 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:43:03.288 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[41141]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:43:03.288 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[41141]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:03.288 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:02 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:03.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:02 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:43:03.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:02 smithi186 ceph-mon[33438]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:43:03.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:02 smithi186 ceph-mon[33438]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:03.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:02 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:03.539 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61421]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:43:03.539 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61421]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:03.539 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61421]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:03.539 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61421]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:03.905 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:43:03.905 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:03.905 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61473]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:03.905 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61473]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:04.167 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61477]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:43:04.167 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61477]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:04.167 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61477]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:04.167 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:43:03 smithi012 sudo[61477]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:04.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61483]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:43:04.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61483]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:04.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61483]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:04.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61483]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 -- 172.21.15.12:0/2980607759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c1020b0 msgr2=0x7f632c1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 --2- 172.21.15.12:0/2980607759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c1020b0 0x7f632c1024d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f631c0099c0 tx=0x7f631c009cd0 comp rx=0 tx=0).stop 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 -- 172.21.15.12:0/2980607759 shutdown_connections 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 --2- 172.21.15.12:0/2980607759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c1020b0 0x7f632c1024d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 --2- 172.21.15.12:0/2980607759 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c103c50 0x7f632c10a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 --2- 172.21.15.12:0/2980607759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c1032b0 0x7f632c103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 -- 172.21.15.12:0/2980607759 >> 172.21.15.12:0/2980607759 conn(0x7f632c0fd690 msgr2=0x7f632c0ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:43:04.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 -- 172.21.15.12:0/2980607759 shutdown_connections 2024-03-26T04:43:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.661+0000 7f6330dd8700 1 -- 172.21.15.12:0/2980607759 wait complete. 2024-03-26T04:43:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 Processor -- start 2024-03-26T04:43:04.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 -- start start 2024-03-26T04:43:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 0x7f632c197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:04.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c1032b0 0x7f632c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c103c50 0x7f632c19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19f190 con 0x7f632c103c50 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19f2d0 con 0x7f632c1032b0 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6330dd8700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19f460 con 0x7f632c1020b0 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632a59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 0x7f632c197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c103c50 0x7f632c19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632a59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 0x7f632c197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:54640/0 (socket says 172.21.15.12:54640) 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632a59c700 1 -- 172.21.15.12:0/3360812564 learned_addr learned my addr 172.21.15.12:0/3360812564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f6329d9b700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c1032b0 0x7f632c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 -- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 msgr2=0x7f632c197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 0x7f632c197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 -- 172.21.15.12:0/3360812564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c1032b0 msgr2=0x7f632c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c1032b0 0x7f632c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 -- 172.21.15.12:0/3360812564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009c40 con 0x7f632c103c50 2024-03-26T04:43:04.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632a59c700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 0x7f632c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.662+0000 7f632ad9d700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c103c50 0x7f632c19ebc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f632400b4a0 tx=0x7f632400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.663+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324015070 con 0x7f632c103c50 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.663+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c0095f0 con 0x7f632c103c50 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.663+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6324003a00 con 0x7f632c103c50 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.663+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324004530 con 0x7f632c103c50 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.663+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c074630 con 0x7f632c103c50 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.664+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c109590 con 0x7f632c103c50 2024-03-26T04:43:04.666 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.664+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 14) v1 ==== 91342+0+0 (secure 0 0 0) 0x7f6324004690 con 0x7f632c103c50 2024-03-26T04:43:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.665+0000 7f631b7fe700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f631406d940 0x7f631406fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.665+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3017+0+0 (secure 0 0 0) 0x7f632408fea0 con 0x7f632c103c50 2024-03-26T04:43:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.665+0000 7f632a59c700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f631406d940 0x7f631406fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.665+0000 7f632a59c700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f631406d940 0x7f631406fe00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f631c000c00 tx=0x7f631c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:43:04.670 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.670+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f632405a520 con 0x7f632c103c50 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi012 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi012 to 5356M 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:04.700 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 ceph-mon[33438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36029]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:04.701 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36029]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:04.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.883+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f632c0624c0 con 0x7f631406d940 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi012 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi012 to 5356M 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.884 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[41141]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61506]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61506]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61506]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 sudo[61506]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi012 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi012 to 5356M 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:04.886 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:04 smithi012 ceph-mon[29863]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:04.949 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:04.949+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f6324093030 con 0x7f632c103c50 2024-03-26T04:43:04.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:43:04.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:04.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36032]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:04.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:04 smithi186 sudo[36032]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:05.490 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61603]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:43:05.490 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61603]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:05.490 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61603]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:05.490 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61603]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61660]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61660]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 sudo[61660]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.817 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:05.818 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:05.819 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.820 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:05.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:43:05.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:43:06.630 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:06 smithi012 ceph-mon[29863]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:06.630 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:06 smithi012 ceph-mon[29863]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:06.630 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:06 smithi012 ceph-mon[29863]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:06.631 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:06 smithi012 ceph-mon[41141]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:06.631 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:06 smithi012 ceph-mon[41141]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:06.631 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:06 smithi012 ceph-mon[41141]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:06.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:06 smithi186 ceph-mon[33438]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:06.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:06 smithi186 ceph-mon[33438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:43:06.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:06 smithi186 ceph-mon[33438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:43:07.810 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[29863]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:07.810 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/367684459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec8405e3-d836-443e-98d6-998c0ceb0a2f"}]: dispatch 2024-03-26T04:43:07.810 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/367684459' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec8405e3-d836-443e-98d6-998c0ceb0a2f"}]': finished 2024-03-26T04:43:07.810 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[29863]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T04:43:07.810 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:07.810 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[41141]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:07.811 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/367684459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec8405e3-d836-443e-98d6-998c0ceb0a2f"}]: dispatch 2024-03-26T04:43:07.811 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/367684459' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec8405e3-d836-443e-98d6-998c0ceb0a2f"}]': finished 2024-03-26T04:43:07.811 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[41141]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T04:43:07.811 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:07 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:07.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:07 smithi186 ceph-mon[33438]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:07.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/367684459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec8405e3-d836-443e-98d6-998c0ceb0a2f"}]: dispatch 2024-03-26T04:43:07.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/367684459' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec8405e3-d836-443e-98d6-998c0ceb0a2f"}]': finished 2024-03-26T04:43:07.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:07 smithi186 ceph-mon[33438]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T04:43:07.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:07 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:08.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1857887983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:08.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1857887983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:08.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1857887983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:09.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:09 smithi012 ceph-mon[41141]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:09.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:09 smithi012 ceph-mon[29863]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:09.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:09 smithi186 ceph-mon[33438]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:10.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:10 smithi186 ceph-mon[33438]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:10.971 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:10 smithi012 ceph-mon[29863]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:10.971 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:10 smithi012 ceph-mon[41141]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:12.799 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:12 smithi012 ceph-mon[29863]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:12.799 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:12 smithi012 ceph-mon[41141]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:12.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:12 smithi186 ceph-mon[33438]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:14.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:43:14.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:43:14.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:14.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:43:14.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:14 smithi186 ceph-mon[33438]: Deploying daemon osd.3 on smithi012 2024-03-26T04:43:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:14 smithi186 ceph-mon[33438]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:15.359 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:14 smithi012 ceph-mon[29863]: Deploying daemon osd.3 on smithi012 2024-03-26T04:43:15.359 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:14 smithi012 ceph-mon[29863]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:15.359 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:14 smithi012 ceph-mon[41141]: Deploying daemon osd.3 on smithi012 2024-03-26T04:43:15.359 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:14 smithi012 ceph-mon[41141]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:16.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:16 smithi012 ceph-mon[29863]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:16.891 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:16 smithi012 ceph-mon[41141]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:16.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:16 smithi186 ceph-mon[33438]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:18.759 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:18 smithi012 ceph-mon[29863]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:18.760 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:18 smithi012 ceph-mon[41141]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:18.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:18 smithi186 ceph-mon[33438]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:20.518 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:20 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:20.518 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:20 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:20.518 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:20 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:20.518 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:20 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:20.518 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:20 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:20.518 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:20 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:20.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:20.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:20.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:20 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:21.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:21 smithi012 ceph-mon[29863]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:21.532 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:21 smithi012 ceph-mon[41141]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:21.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:21 smithi186 ceph-mon[33438]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:22.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.984+0000 7f631b7fe700 1 -- 172.21.15.12:0/3360812564 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f632c0624c0 con 0x7f631406d940 2024-03-26T04:43:22.985 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 3 on host 'smithi012' 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f631406d940 msgr2=0x7f631406fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f631406d940 0x7f631406fe00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f631c000c00 tx=0x7f631c005fb0 comp rx=0 tx=0).stop 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c103c50 msgr2=0x7f632c19ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c103c50 0x7f632c19ebc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f632400b4a0 tx=0x7f632400b7b0 comp rx=0 tx=0).stop 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 shutdown_connections 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f631406d940 0x7f631406fe00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f632c103c50 0x7f632c19ebc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f632c1020b0 0x7f632c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 --2- 172.21.15.12:0/3360812564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f632c1032b0 0x7f632c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 >> 172.21.15.12:0/3360812564 conn(0x7f632c0fd690 msgr2=0x7f632c1066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:43:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 shutdown_connections 2024-03-26T04:43:22.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:43:22.986+0000 7f6330dd8700 1 -- 172.21.15.12:0/3360812564 wait complete. 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:23 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:23.681 DEBUG:teuthology.orchestra.run.smithi012:osd.3> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T04:43:23.683 INFO:tasks.cephadm:Deploying osd.4 on smithi186 with /dev/vg_nvme/lv_4... 2024-03-26T04:43:23.684 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:43:23.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:23.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:23 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:23.929 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:43:24.356 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:24 smithi186 ceph-mon[33438]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:43:24.356 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:24 smithi186 ceph-mon[33438]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:43:24.356 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:24 smithi186 ceph-mon[33438]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:43:24.356 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:24 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:24.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[29863]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:43:24.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[29863]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:43:24.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[29863]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:43:24.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:24.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[41141]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:43:24.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[41141]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:43:24.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[41141]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-03-26T04:43:24.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:24.656 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3[63439]: 2024-03-26T04:43:24.277+0000 7f77c2b03700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T04:43:24.657 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 04:43:24 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3[63439]: 2024-03-26T04:43:24.282+0000 7f77be8f6700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:43:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[29863]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[29863]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[29863]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[41141]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[41141]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:25.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:25 smithi012 ceph-mon[41141]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:25 smithi186 ceph-mon[33438]: from='osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-03-26T04:43:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:25 smithi186 ceph-mon[33438]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:43:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:25 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:25 smithi186 ceph-mon[33438]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:43:25.820 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:43:25.820 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:43:25.820 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-26T04:43:25.820 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-26T04:43:25.820 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117529 s, 892 MB/s 2024-03-26T04:43:25.820 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-26T04:43:26.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:43:26.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:43:26.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[29863]: osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] boot 2024-03-26T04:43:26.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[29863]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T04:43:26.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:26.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:43:26.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:43:26.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[41141]: osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] boot 2024-03-26T04:43:26.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[41141]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T04:43:26.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:26 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:26.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:26 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:43:26.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:26 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:43:26.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:26 smithi186 ceph-mon[33438]: osd.3 [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] boot 2024-03-26T04:43:26.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:26 smithi186 ceph-mon[33438]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T04:43:26.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:26 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:43:26.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-03-26T04:43:26.810 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:43:27.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi012 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi012 to 4017M 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:27.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:27.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:27 smithi186 ceph-mon[33438]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi012 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi012 to 4017M 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[41141]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi012 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi012 to 4017M 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:27 smithi012 ceph-mon[29863]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/207964704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1032b0 msgr2=0x7fa9cc103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/207964704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1032b0 0x7fa9cc103710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40099c0 tx=0x7fa9b4009cd0 comp rx=0 tx=0).stop 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/207964704 shutdown_connections 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/207964704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/207964704 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc103c50 0x7fa9cc10a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/207964704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1032b0 0x7fa9cc103710 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:28.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/207964704 >> 172.21.15.186:0/207964704 conn(0x7fa9cc0fd690 msgr2=0x7fa9cc0ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:43:28.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/207964704 shutdown_connections 2024-03-26T04:43:28.422 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/207964704 wait complete. 2024-03-26T04:43:28.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.420+0000 7fa9d0ef8700 1 Processor -- start 2024-03-26T04:43:28.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 -- start start 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1032b0 0x7fa9cc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc103c50 0x7fa9cc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc19f160 con 0x7fa9cc1032b0 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc19f2a0 con 0x7fa9cc1020b0 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9d0ef8700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc19f430 con 0x7fa9cc103c50 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39662/0 (socket says 172.21.15.186:39662) 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9c1d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1032b0 0x7fa9cc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 -- 172.21.15.186:0/3564616851 learned_addr learned my addr 172.21.15.186:0/3564616851 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9cad9d700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc103c50 0x7fa9cc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:28.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 -- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc103c50 msgr2=0x7fa9cc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc103c50 0x7fa9cc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 -- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1032b0 msgr2=0x7fa9cc197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1032b0 0x7fa9cc197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.421+0000 7fa9ca59c700 1 -- 172.21.15.186:0/3564616851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b40095f0 con 0x7fa9cc1020b0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.422+0000 7fa9ca59c700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1975d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00d980 tx=0x7fa9bc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.422+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc004020 con 0x7fa9cc1020b0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.422+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa9bc00f690 con 0x7fa9cc1020b0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.422+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc003980 con 0x7fa9cc1020b0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.422+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc19f710 con 0x7fa9cc1020b0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.422+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc19fc10 con 0x7fa9cc1020b0 2024-03-26T04:43:28.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.423+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9cc109530 con 0x7fa9cc1020b0 2024-03-26T04:43:28.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.423+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7fa9bc004180 con 0x7fa9cc1020b0 2024-03-26T04:43:28.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.424+0000 7fa9c3fff700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa9b806d9d0 0x7fa9b806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:28.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.424+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3449+0+0 (secure 0 0 0) 0x7fa9bc08c340 con 0x7fa9cc1020b0 2024-03-26T04:43:28.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.424+0000 7fa9c1d9b700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa9b806d9d0 0x7fa9b806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:28.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.424+0000 7fa9c1d9b700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa9b806d9d0 0x7fa9b806fe90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b580 tx=0x7fa9b4005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:43:28.429 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.429+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa9bc056730 con 0x7fa9cc1020b0 2024-03-26T04:43:28.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:28.628+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9cc0624c0 con 0x7fa9b806d9d0 2024-03-26T04:43:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:28 smithi186 ceph-mon[33438]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:28 smithi186 ceph-mon[33438]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[41141]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[41141]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[29863]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[29863]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:30.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:30 smithi186 ceph-mon[33438]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:31.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:30 smithi012 ceph-mon[41141]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:30 smithi012 ceph-mon[29863]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:31.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/3049875113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]: dispatch 2024-03-26T04:43:31.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]: dispatch 2024-03-26T04:43:31.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]': finished 2024-03-26T04:43:31.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:31 smithi186 ceph-mon[33438]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T04:43:31.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:31 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:32.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/3049875113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]: dispatch 2024-03-26T04:43:32.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]: dispatch 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]': finished 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[41141]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/3049875113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]: dispatch 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]: dispatch 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3"}]': finished 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[29863]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T04:43:32.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:31 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:32.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/891277093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:32.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:32 smithi186 ceph-mon[33438]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:33.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/891277093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:33.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:32 smithi012 ceph-mon[29863]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:33.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/891277093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:33.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:32 smithi012 ceph-mon[41141]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:34.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:34 smithi186 ceph-mon[33438]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:35.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:34 smithi012 ceph-mon[29863]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:35.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:34 smithi012 ceph-mon[41141]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:36.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:36 smithi186 ceph-mon[33438]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:37.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:36 smithi012 ceph-mon[29863]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:37.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:36 smithi012 ceph-mon[41141]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:37.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:37 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:43:37.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:37 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:38.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:37 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:43:38.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:37 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:38.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:37 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:43:38.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:37 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:38.910 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:38 smithi186 ceph-mon[33438]: Deploying daemon osd.4 on smithi186 2024-03-26T04:43:38.910 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:38 smithi186 ceph-mon[33438]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:39.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:38 smithi012 ceph-mon[29863]: Deploying daemon osd.4 on smithi186 2024-03-26T04:43:39.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:38 smithi012 ceph-mon[29863]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:39.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:38 smithi012 ceph-mon[41141]: Deploying daemon osd.4 on smithi186 2024-03-26T04:43:39.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:38 smithi012 ceph-mon[41141]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:40.926 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:40 smithi186 ceph-mon[33438]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:41.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:40 smithi012 ceph-mon[29863]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:41.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:40 smithi012 ceph-mon[41141]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:42.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:42 smithi186 ceph-mon[33438]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:43.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:42 smithi012 ceph-mon[29863]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:43.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:42 smithi012 ceph-mon[41141]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:44.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:44.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:44.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:44.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:44.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:44.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:44.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:44.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:44.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:45.071 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:44 smithi186 ceph-mon[33438]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:45.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:44 smithi012 ceph-mon[29863]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:45.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:44 smithi012 ceph-mon[41141]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:46.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.642+0000 7fa9c3fff700 1 -- 172.21.15.186:0/3564616851 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa9cc0624c0 con 0x7fa9b806d9d0 2024-03-26T04:43:46.643 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 4 on host 'smithi186' 2024-03-26T04:43:46.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa9b806d9d0 msgr2=0x7fa9b806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:46.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa9b806d9d0 0x7fa9b806fe90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b580 tx=0x7fa9b4005e80 comp rx=0 tx=0).stop 2024-03-26T04:43:46.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 msgr2=0x7fa9cc1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:46.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1975d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00d980 tx=0x7fa9bc00dc90 comp rx=0 tx=0).stop 2024-03-26T04:43:46.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 shutdown_connections 2024-03-26T04:43:46.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fa9b806d9d0 0x7fa9b806fe90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:46.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fa9cc103c50 0x7fa9cc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:46.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9cc1020b0 0x7fa9cc1975d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:46.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 --2- 172.21.15.186:0/3564616851 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa9cc1032b0 0x7fa9cc197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:46.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 >> 172.21.15.186:0/3564616851 conn(0x7fa9cc0fd690 msgr2=0x7fa9cc106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:43:46.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 shutdown_connections 2024-03-26T04:43:46.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:46.644+0000 7fa9d0ef8700 1 -- 172.21.15.186:0/3564616851 wait complete. 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:46.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:46.939 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:47.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:47.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:47.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:47.272 DEBUG:teuthology.orchestra.run.smithi186:osd.4> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T04:43:47.275 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-03-26T04:43:47.275 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:43:47.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-mon[33438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:43:47.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-mon[33438]: from='osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:43:47.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-mon[33438]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:43:47.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:47.750 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-mon[33438]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:43:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[29863]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:43:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[29863]: from='osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:43:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[29863]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:43:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[29863]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:43:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[41141]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:43:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[41141]: from='osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:43:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[41141]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:43:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:47 smithi012 ceph-mon[41141]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:43:48.161 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T04:43:47.749+0000 7fe46b780700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T04:43:48.161 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 04:43:47 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T04:43:47.753+0000 7fe467573700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:43:48.540 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] boot 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T04:43:48.772 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:48 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:43:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] boot 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: osd.4 [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] boot 2024-03-26T04:43:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T04:43:49.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:48 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:43:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:49 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:43:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:49 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:43:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:49 smithi186 ceph-mon[33438]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:43:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:49 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:43:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:49 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:43:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:49 smithi012 ceph-mon[29863]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:43:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:49 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:43:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:49 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:43:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:49 smithi012 ceph-mon[41141]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:43:51.064 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:43:51.064 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:43:51.064 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-26T04:43:51.065 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-26T04:43:51.065 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114295 s, 917 MB/s 2024-03-26T04:43:51.065 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-26T04:43:51.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:51.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:51 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.676 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-03-26T04:43:51.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:43:51.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:43:51.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:51.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:51.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:51 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:43:51.910 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:43:52.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:52 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi186 2024-03-26T04:43:52.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:52 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi186 to 17088M 2024-03-26T04:43:52.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:52 smithi186 ceph-mon[33438]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:43:52.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:52 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi186 2024-03-26T04:43:52.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:52 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi186 to 17088M 2024-03-26T04:43:52.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:52 smithi012 ceph-mon[29863]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:43:52.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:52 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi186 2024-03-26T04:43:52.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:52 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi186 to 17088M 2024-03-26T04:43:52.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:52 smithi012 ceph-mon[41141]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 -- 172.21.15.186:0/719874065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8100fb0 msgr2=0x7f37f81013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/719874065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8100fb0 0x7f37f81013d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f37e80099c0 tx=0x7f37e8009cd0 comp rx=0 tx=0).stop 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 -- 172.21.15.186:0/719874065 shutdown_connections 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/719874065 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37f8102b50 0x7f37f8109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/719874065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8100fb0 0x7f37f81013d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/719874065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f81021b0 0x7f37f8102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 -- 172.21.15.186:0/719874065 >> 172.21.15.186:0/719874065 conn(0x7f37f80fc570 msgr2=0x7f37f80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:43:53.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 -- 172.21.15.186:0/719874065 shutdown_connections 2024-03-26T04:43:53.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.514+0000 7f37fdcf6700 1 -- 172.21.15.186:0/719874065 wait complete. 2024-03-26T04:43:53.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 Processor -- start 2024-03-26T04:43:53.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 -- start start 2024-03-26T04:43:53.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37f8100fb0 0x7f37f8104ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:53.517 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f81021b0 0x7f37f8103590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 0x7f37f8103ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f8104270 con 0x7f37f81021b0 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f81043e0 con 0x7f37f8102b50 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.515+0000 7f37fdcf6700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f81a1700 con 0x7f37f8100fb0 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 0x7f37f8103ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 0x7f37f8103ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42230/0 (socket says 172.21.15.186:42230) 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 -- 172.21.15.186:0/50949893 learned_addr learned my addr 172.21.15.186:0/50949893 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f77fe700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37f8100fb0 0x7f37f8104ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f6ffd700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f81021b0 0x7f37f8103590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 -- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37f8100fb0 msgr2=0x7f37f8104ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37f8100fb0 0x7f37f8104ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 -- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f81021b0 msgr2=0x7f37f8103590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:43:53.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f81021b0 0x7f37f8103590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 -- 172.21.15.186:0/50949893 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e80095f0 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f7fff700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 0x7f37f8103ad0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37f000ada0 tx=0x7f37f000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f000f070 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f81a1990 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f37f0007b30 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0005280 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.516+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f81a1ee0 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.517+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f8108d90 con 0x7f37f8102b50 2024-03-26T04:43:53.519 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.518+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f37f000c8d0 con 0x7f37f8102b50 2024-03-26T04:43:53.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.518+0000 7f37f4ff9700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f37e006d9d0 0x7f37e006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:43:53.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.518+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 3991+0+0 (secure 0 0 0) 0x7f37f008bf60 con 0x7f37f8102b50 2024-03-26T04:43:53.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.518+0000 7f37f77fe700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f37e006d9d0 0x7f37e006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:43:53.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.519+0000 7f37f77fe700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f37e006d9d0 0x7f37e006fe90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f37e800c010 tx=0x7f37e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:43:53.522 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.523+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f37f00561b0 con 0x7f37f8102b50 2024-03-26T04:43:53.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:53 smithi186 ceph-mon[33438]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:43:53.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:53 smithi186 ceph-mon[33438]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-03-26T04:43:53.726 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:43:53.727+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f37f80624c0 con 0x7f37e006d9d0 2024-03-26T04:43:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:53 smithi012 ceph-mon[29863]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:43:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:53 smithi012 ceph-mon[29863]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-03-26T04:43:53.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:53 smithi012 ceph-mon[41141]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:43:53.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:53 smithi012 ceph-mon[41141]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 149 KiB/s, 0 objects/s recovering 2024-03-26T04:43:54.663 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:54 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:54.663 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:54 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:54.664 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:54 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:54.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:54 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:54.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:54 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:54.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:54 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:54.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:54 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:43:54.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:54 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:43:54.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:54 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:55.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:55 smithi186 ceph-mon[33438]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:55.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:55 smithi186 ceph-mon[33438]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-26T04:43:55.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:55 smithi012 ceph-mon[29863]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:55.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:55 smithi012 ceph-mon[29863]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-26T04:43:55.790 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:55 smithi012 ceph-mon[41141]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:55.790 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:55 smithi012 ceph-mon[41141]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-26T04:43:56.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/1670516892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]: dispatch 2024-03-26T04:43:56.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]: dispatch 2024-03-26T04:43:56.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]': finished 2024-03-26T04:43:56.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[29863]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T04:43:56.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:43:56.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/1670516892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]: dispatch 2024-03-26T04:43:56.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]: dispatch 2024-03-26T04:43:56.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]': finished 2024-03-26T04:43:56.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[41141]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T04:43:56.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:56 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:43:56.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/1670516892' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]: dispatch 2024-03-26T04:43:56.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]: dispatch 2024-03-26T04:43:56.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f918fca-3303-4d6c-a0d5-81d952478987"}]': finished 2024-03-26T04:43:56.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:56 smithi186 ceph-mon[33438]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T04:43:56.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:56 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:43:57.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:57 smithi012 ceph-mon[29863]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:43:57.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/2279213251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:57.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:57 smithi012 ceph-mon[41141]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:43:57.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/2279213251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:57.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:57 smithi186 ceph-mon[33438]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:43:57.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/2279213251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:43:58.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:43:58 smithi186 ceph-mon[33438]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:43:59.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:43:58 smithi012 ceph-mon[29863]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:43:59.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:43:58 smithi012 ceph-mon[41141]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:00.886 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:00 smithi186 ceph-mon[33438]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:01.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:00 smithi012 ceph-mon[29863]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:01.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:00 smithi012 ceph-mon[41141]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:02.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:02 smithi186 ceph-mon[33438]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:03.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:02 smithi012 ceph-mon[29863]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:03.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:02 smithi012 ceph-mon[41141]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:03 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:44:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:03 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:03 smithi186 ceph-mon[33438]: Deploying daemon osd.5 on smithi186 2024-03-26T04:44:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:03 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:44:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:03 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:03 smithi012 ceph-mon[29863]: Deploying daemon osd.5 on smithi186 2024-03-26T04:44:04.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:03 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:44:04.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:03 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:04.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:03 smithi012 ceph-mon[41141]: Deploying daemon osd.5 on smithi186 2024-03-26T04:44:04.827 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:04 smithi186 ceph-mon[33438]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:05.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:04 smithi012 ceph-mon[29863]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:05.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:04 smithi012 ceph-mon[41141]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:06.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:06 smithi186 ceph-mon[33438]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:07.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:06 smithi012 ceph-mon[29863]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:06 smithi012 ceph-mon[41141]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:09.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:09.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:09.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:09.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:09 smithi186 ceph-mon[33438]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:09.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:09.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:09.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:09.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[41141]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:09.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:09.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:09.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:09.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:09 smithi012 ceph-mon[29863]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:10.821 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:10 smithi186 ceph-mon[33438]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:11.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:10 smithi012 ceph-mon[29863]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:11.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:10 smithi012 ceph-mon[41141]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:11.993 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.993+0000 7f37f4ff9700 1 -- 172.21.15.186:0/50949893 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f37f80624c0 con 0x7f37e006d9d0 2024-03-26T04:44:11.993 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-03-26T04:44:11.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f37e006d9d0 msgr2=0x7f37e006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f37e006d9d0 0x7f37e006fe90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f37e800c010 tx=0x7f37e8005fd0 comp rx=0 tx=0).stop 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 msgr2=0x7f37f8103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 0x7f37f8103ad0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37f000ada0 tx=0x7f37f000c330 comp rx=0 tx=0).stop 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 shutdown_connections 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f37e006d9d0 0x7f37e006fe90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37f8102b50 0x7f37f8103ad0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f37f8100fb0 0x7f37f8104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 --2- 172.21.15.186:0/50949893 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f81021b0 0x7f37f8103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 >> 172.21.15.186:0/50949893 conn(0x7f37f80fc570 msgr2=0x7f37f806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.995+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 shutdown_connections 2024-03-26T04:44:11.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:11.996+0000 7f37fdcf6700 1 -- 172.21.15.186:0/50949893 wait complete. 2024-03-26T04:44:12.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:12 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:12 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:12.613 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T04:44:12.615 INFO:tasks.cephadm:Deploying osd.6 on smithi186 with /dev/vg_nvme/lv_2... 2024-03-26T04:44:12.616 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:44:13.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-mon[33438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:44:13.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-mon[33438]: from='osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:13.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-mon[33438]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:44:13.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:13.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-mon[33438]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:13.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-mon[33438]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:13.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[29863]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:44:13.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[29863]: from='osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[29863]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[29863]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[29863]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[41141]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[41141]: from='osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[41141]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[41141]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:13.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:13 smithi012 ceph-mon[41141]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:13.858 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:44:13.964 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T04:44:13.616+0000 7f90797c0700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T04:44:13.964 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 04:44:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T04:44:13.620+0000 7f90715ab700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:44:14.311 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:14 smithi186 ceph-mon[33438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:44:14.311 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:14 smithi186 ceph-mon[33438]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:44:14.311 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:14 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:14.311 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:14 smithi186 ceph-mon[33438]: from='osd.5 ' entity='osd.5' 2024-03-26T04:44:14.311 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:14 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[29863]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[29863]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[29863]: from='osd.5 ' entity='osd.5' 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[41141]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[41141]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:44:14.557 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:14.558 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[41141]: from='osd.5 ' entity='osd.5' 2024-03-26T04:44:14.558 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:14 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:15.411 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:44:15.412 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:44:15.412 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] boot 2024-03-26T04:44:15.412 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T04:44:15.412 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:15.412 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:44:15.412 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:15 smithi186 ceph-mon[33438]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] boot 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[29863]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:44:15.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:44:15.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: osd.5 [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] boot 2024-03-26T04:44:15.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T04:44:15.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:44:15.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:44:15.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:15 smithi012 ceph-mon[41141]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:44:16.235 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:44:16.235 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:44:16.235 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-26T04:44:16.235 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-26T04:44:16.235 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122799 s, 854 MB/s 2024-03-26T04:44:16.235 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-26T04:44:16.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:16 smithi012 ceph-mon[29863]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:44:16.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:16 smithi012 ceph-mon[41141]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:44:16.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:16 smithi186 ceph-mon[33438]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:44:17.308 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-03-26T04:44:17.546 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:44:17.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:44:17.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:17.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:44:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:17.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:17.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:18.703 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:18 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi186 2024-03-26T04:44:18.703 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:18 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi186 to 8544M 2024-03-26T04:44:18.703 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:18 smithi186 ceph-mon[33438]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:44:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:18 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi186 2024-03-26T04:44:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:18 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi186 to 8544M 2024-03-26T04:44:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:18 smithi012 ceph-mon[29863]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:44:18.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:18 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi186 2024-03-26T04:44:18.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:18 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi186 to 8544M 2024-03-26T04:44:18.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:18 smithi012 ceph-mon[41141]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.228+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3898431411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102220 msgr2=0x7f99bc102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.228+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3898431411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102220 0x7f99bc102680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0099c0 tx=0x7f99ac009cd0 comp rx=0 tx=0).stop 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.228+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3898431411 shutdown_connections 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.228+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3898431411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99bc102bc0 0x7f99bc109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.228+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3898431411 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99bc101020 0x7f99bc101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.228+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3898431411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102220 0x7f99bc102680 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.229+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3898431411 >> 172.21.15.186:0/3898431411 conn(0x7f99bc0fc5e0 msgr2=0x7f99bc0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:44:19.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.229+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3898431411 shutdown_connections 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.229+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3898431411 wait complete. 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.229+0000 7f99c0cb1700 1 Processor -- start 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.229+0000 7f99c0cb1700 1 -- start start 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.229+0000 7f99c0cb1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99bc101020 0x7f99bc197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99bc102220 0x7f99bc197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 0x7f99bc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc19f200 con 0x7f99bc102220 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc19f340 con 0x7f99bc102bc0 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc19f4d0 con 0x7f99bc101020 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 0x7f99bc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 0x7f99bc19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34492/0 (socket says 172.21.15.186:34492) 2024-03-26T04:44:19.232 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 -- 172.21.15.186:0/3902314837 learned_addr learned my addr 172.21.15.186:0/3902314837 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99b9d9b700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99bc102220 0x7f99bc197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99ba59c700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99bc101020 0x7f99bc197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 -- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99bc101020 msgr2=0x7f99bc197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99bc101020 0x7f99bc197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 -- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99bc102220 msgr2=0x7f99bc197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99bc102220 0x7f99bc197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 -- 172.21.15.186:0/3902314837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac0095f0 con 0x7f99bc102bc0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99bad9d700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 0x7f99bc19ec30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99b400fe00 tx=0x7f99b400fe30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4010c90 con 0x7f99bc102bc0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc109e20 con 0x7f99bc102bc0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f99b4010df0 con 0x7f99bc102bc0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b40075b0 con 0x7f99bc102bc0 2024-03-26T04:44:19.233 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.230+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc10a350 con 0x7f99bc102bc0 2024-03-26T04:44:19.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.232+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99bc107c00 con 0x7f99bc102bc0 2024-03-26T04:44:19.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.232+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f99b4018440 con 0x7f99bc102bc0 2024-03-26T04:44:19.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.232+0000 7f99ab7fe700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f99a406d9d0 0x7f99a406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:19.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.232+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4423+0+0 (secure 0 0 0) 0x7f99b4006e40 con 0x7f99bc102bc0 2024-03-26T04:44:19.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.233+0000 7f99ba59c700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f99a406d9d0 0x7f99a406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:19.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.233+0000 7f99ba59c700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f99a406d9d0 0x7f99a406fe90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f99bc102080 tx=0x7f99b0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:44:19.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.237+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f99b405a680 con 0x7f99bc102bc0 2024-03-26T04:44:19.437 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:19.437+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f99bc10d200 con 0x7f99a406d9d0 2024-03-26T04:44:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:19 smithi186 ceph-mon[33438]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:19.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:44:19.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:44:19.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:19.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[29863]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:19.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:44:19.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:44:19.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:19.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[41141]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:19.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:44:19.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:44:19.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:20.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:20 smithi186 ceph-mon[33438]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:20 smithi012 ceph-mon[29863]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:20.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:20 smithi012 ceph-mon[41141]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:21.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:21 smithi186 ceph-mon[33438]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:21.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:21 smithi012 ceph-mon[29863]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:21.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:21 smithi012 ceph-mon[41141]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:22.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/417132752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]: dispatch 2024-03-26T04:44:22.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]: dispatch 2024-03-26T04:44:22.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]': finished 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[29863]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/417132752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]: dispatch 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]: dispatch 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]': finished 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[41141]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T04:44:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:22 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:22.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/417132752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]: dispatch 2024-03-26T04:44:22.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]: dispatch 2024-03-26T04:44:22.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdc95ff2-55b2-4401-b0d7-d8b58beb916d"}]': finished 2024-03-26T04:44:22.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:22 smithi186 ceph-mon[33438]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T04:44:22.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:22 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:23.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:23 smithi012 ceph-mon[29863]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:23.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/536537658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:44:23.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:23 smithi012 ceph-mon[41141]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:23.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/536537658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:44:23.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:23 smithi186 ceph-mon[33438]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:23.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/536537658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:44:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:24 smithi186 ceph-mon[33438]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:24 smithi012 ceph-mon[29863]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:25.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:24 smithi012 ceph-mon[41141]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:26 smithi186 ceph-mon[33438]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:27.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:26 smithi012 ceph-mon[29863]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:27.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:26 smithi012 ceph-mon[41141]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:28 smithi186 ceph-mon[33438]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:29.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:28 smithi012 ceph-mon[29863]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:28 smithi012 ceph-mon[41141]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:29.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:29 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:44:29.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:29 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:29.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:29 smithi186 ceph-mon[33438]: Deploying daemon osd.6 on smithi186 2024-03-26T04:44:30.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:29 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:44:30.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:29 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:30.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:29 smithi012 ceph-mon[29863]: Deploying daemon osd.6 on smithi186 2024-03-26T04:44:30.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:29 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:44:30.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:29 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:30.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:29 smithi012 ceph-mon[41141]: Deploying daemon osd.6 on smithi186 2024-03-26T04:44:30.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:30 smithi186 ceph-mon[33438]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:30 smithi012 ceph-mon[29863]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:31.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:30 smithi012 ceph-mon[41141]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:32.782 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:32 smithi186 ceph-mon[33438]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:33.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:32 smithi012 ceph-mon[29863]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:33.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:32 smithi012 ceph-mon[41141]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:35.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:35.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:35.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:35.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:35 smithi186 ceph-mon[33438]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:35.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:35.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:35.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:35.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[29863]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:35.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:35.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:35.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:35.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:35 smithi012 ceph-mon[41141]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:36.722 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:36 smithi186 ceph-mon[33438]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:37.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:36 smithi012 ceph-mon[29863]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:37.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:36 smithi012 ceph-mon[41141]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:37.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:37 smithi186 ceph-mon[33438]: from='osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:44:37.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:37 smithi186 ceph-mon[33438]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:44:38.101 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.101+0000 7f99ab7fe700 1 -- 172.21.15.186:0/3902314837 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f99bc10d200 con 0x7f99a406d9d0 2024-03-26T04:44:38.101 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 6 on host 'smithi186' 2024-03-26T04:44:38.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.102+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f99a406d9d0 msgr2=0x7f99a406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:38.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.102+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f99a406d9d0 0x7f99a406fe90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f99bc102080 tx=0x7f99b0009190 comp rx=0 tx=0).stop 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.102+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 msgr2=0x7f99bc19ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.102+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 0x7f99bc19ec30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f99b400fe00 tx=0x7f99b400fe30 comp rx=0 tx=0).stop 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 shutdown_connections 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f99a406d9d0 0x7f99a406fe90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99bc102bc0 0x7f99bc19ec30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99bc101020 0x7f99bc197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 --2- 172.21.15.186:0/3902314837 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99bc102220 0x7f99bc197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 >> 172.21.15.186:0/3902314837 conn(0x7f99bc0fc5e0 msgr2=0x7f99bc0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 shutdown_connections 2024-03-26T04:44:38.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:38.103+0000 7f99c0cb1700 1 -- 172.21.15.186:0/3902314837 wait complete. 2024-03-26T04:44:38.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:37 smithi012 ceph-mon[29863]: from='osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:44:38.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:37 smithi012 ceph-mon[29863]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:44:38.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:37 smithi012 ceph-mon[41141]: from='osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:44:38.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:37 smithi012 ceph-mon[41141]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:44:38.723 DEBUG:teuthology.orchestra.run.smithi186:osd.6> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T04:44:38.726 INFO:tasks.cephadm:Deploying osd.7 on smithi186 with /dev/vg_nvme/lv_1... 2024-03-26T04:44:38.726 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:44:38.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: from='osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:38.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:38 smithi186 ceph-mon[33438]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:39.023 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:44:39.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:44:39.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: from='osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:39.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[29863]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: from='osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:39.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:39.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:38 smithi012 ceph-mon[41141]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:44:39.403 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T04:44:39.180+0000 7f53c0473700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T04:44:39.403 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T04:44:39.186+0000 7f53b7a5d700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:44:39.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='osd.6 ' entity='osd.6' 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] boot 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T04:44:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:40.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='osd.6 ' entity='osd.6' 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] boot 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:40.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='osd.6 ' entity='osd.6' 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: osd.6 [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] boot 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T04:44:40.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:44:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:41 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:44:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:41 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:44:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:41 smithi186 ceph-mon[33438]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:44:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:41 smithi186 ceph-mon[33438]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:41.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:44:41.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:44:41.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[29863]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:44:41.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[29863]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:41.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:44:41.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:44:41.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[41141]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:44:41.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:41 smithi012 ceph-mon[41141]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:42.480 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:44:42.480 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:44:42.480 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-26T04:44:42.480 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-26T04:44:42.480 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118746 s, 883 MB/s 2024-03-26T04:44:42.480 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-26T04:44:42.724 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:42 smithi186 ceph-mon[33438]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:44:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:42 smithi012 ceph-mon[29863]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:44:42.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:42 smithi012 ceph-mon[41141]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:44:43.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:44:43.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:43.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.873 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: pgmap v158: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:44:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:44.111 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi186 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi186 to 5696M 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:44.807 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:44:44.617+0000 7f0560d2c700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi186 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi186 to 5696M 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:44.808 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi186 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi186 to 5696M 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:44.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:44:45.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:45 smithi186 ceph-mon[33438]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-26T04:44:45.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 -- 172.21.15.186:0/1431301905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 msgr2=0x7f8078105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:45.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 --2- 172.21.15.186:0/1431301905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f8078105730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8068009990 tx=0x7f8068009ca0 comp rx=0 tx=0).stop 2024-03-26T04:44:45.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 -- 172.21.15.186:0/1431301905 shutdown_connections 2024-03-26T04:44:45.785 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 --2- 172.21.15.186:0/1431301905 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078105c70 0x7f80781060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 --2- 172.21.15.186:0/1431301905 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078100930 0x7f8078102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 --2- 172.21.15.186:0/1431301905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f8078105730 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 -- 172.21.15.186:0/1431301905 >> 172.21.15.186:0/1431301905 conn(0x7f80780fc570 msgr2=0x7f80780fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 -- 172.21.15.186:0/1431301905 shutdown_connections 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.785+0000 7f80801d4700 1 -- 172.21.15.186:0/1431301905 wait complete. 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 Processor -- start 2024-03-26T04:44:45.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 -- start start 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078100930 0x7f807819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f807819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078105c70 0x7f80781a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80781a5750 con 0x7f8078100930 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80781a5890 con 0x7f80781032a0 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f80801d4700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80781a5a20 con 0x7f8078105c70 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f807819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f807819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43956/0 (socket says 172.21.15.186:43956) 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 -- 172.21.15.186:0/3761983346 learned_addr learned my addr 172.21.15.186:0/3761983346 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807df70700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078100930 0x7f807819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:45.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807e771700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078105c70 0x7f80781a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 -- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078105c70 msgr2=0x7f80781a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078105c70 0x7f80781a5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 -- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078100930 msgr2=0x7f807819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078100930 0x7f807819dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 -- 172.21.15.186:0/3761983346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80680095f0 con 0x7f80781032a0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807df70700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078100930 0x7f807819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807e771700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078105c70 0x7f80781a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f807d76f700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f807819e100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f80680098e0 tx=0x7f806800f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.786+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806801d070 con 0x7f80781032a0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.787+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8068022470 con 0x7f80781032a0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.787+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068003a20 con 0x7f80781032a0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.787+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80781a5ca0 con 0x7f80781032a0 2024-03-26T04:44:45.789 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.787+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80781a60b0 con 0x7f80781032a0 2024-03-26T04:44:45.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.788+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80780fe0f0 con 0x7f80781032a0 2024-03-26T04:44:45.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.788+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f806800fb70 con 0x7f80781032a0 2024-03-26T04:44:45.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.788+0000 7f806effd700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f806406d9d0 0x7f806406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:44:45.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.788+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4855+0+0 (secure 0 0 0) 0x7f806808e4d0 con 0x7f80781032a0 2024-03-26T04:44:45.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.789+0000 7f807df70700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f806406d9d0 0x7f806406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:44:45.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.789+0000 7f807df70700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f806406d9d0 0x7f806406fe90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8074005950 tx=0x7f8074005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:44:45.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.793+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8068058460 con 0x7f80781032a0 2024-03-26T04:44:45.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:45 smithi012 ceph-mon[29863]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-26T04:44:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:45 smithi012 ceph-mon[41141]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-26T04:44:45.995 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:44:45.995+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f80780624c0 con 0x7f806406d9d0 2024-03-26T04:44:46.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:44:46.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:44:46.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:46 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:44:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:44:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:46 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:44:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:44:46.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:46 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:47.764 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:47 smithi186 ceph-mon[33438]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:47.764 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:47 smithi186 ceph-mon[33438]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:44:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:47 smithi012 ceph-mon[29863]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:47 smithi012 ceph-mon[29863]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:44:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:47 smithi012 ceph-mon[41141]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:47.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:47 smithi012 ceph-mon[41141]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:44:48.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:48 smithi186 ceph-mon[33438]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-26T04:44:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:48 smithi012 ceph-mon[29863]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-26T04:44:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:48 smithi012 ceph-mon[41141]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-26T04:44:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/2819627015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]: dispatch 2024-03-26T04:44:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]: dispatch 2024-03-26T04:44:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]': finished 2024-03-26T04:44:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:49 smithi186 ceph-mon[33438]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T04:44:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:49 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:44:49.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/1567914375' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:44:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/2819627015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]: dispatch 2024-03-26T04:44:50.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]: dispatch 2024-03-26T04:44:50.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]': finished 2024-03-26T04:44:50.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[29863]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T04:44:50.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:44:50.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/1567914375' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:44:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/2819627015' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]: dispatch 2024-03-26T04:44:50.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]: dispatch 2024-03-26T04:44:50.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04551815-3856-4382-85e7-bd7b39e62bd9"}]': finished 2024-03-26T04:44:50.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[41141]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T04:44:50.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:44:50.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/1567914375' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:44:50.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:50 smithi186 ceph-mon[33438]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:51.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:50 smithi012 ceph-mon[29863]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:51.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:50 smithi012 ceph-mon[41141]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:52.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:52 smithi186 ceph-mon[33438]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:53.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:52 smithi012 ceph-mon[29863]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:53.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:52 smithi012 ceph-mon[41141]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:54.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:54 smithi186 ceph-mon[33438]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:55.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:54 smithi012 ceph-mon[29863]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:54 smithi012 ceph-mon[41141]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:55.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:55 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:44:55.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:55 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:55.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:55 smithi186 ceph-mon[33438]: Deploying daemon osd.7 on smithi186 2024-03-26T04:44:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:55 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:44:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:55 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:55 smithi012 ceph-mon[29863]: Deploying daemon osd.7 on smithi186 2024-03-26T04:44:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:55 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:44:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:55 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:56.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:55 smithi012 ceph-mon[41141]: Deploying daemon osd.7 on smithi186 2024-03-26T04:44:56.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:56 smithi186 ceph-mon[33438]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:56 smithi012 ceph-mon[29863]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:56 smithi012 ceph-mon[41141]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:58.913 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:44:58 smithi186 ceph-mon[33438]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:59.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:44:58 smithi012 ceph-mon[41141]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:44:59.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:44:58 smithi012 ceph-mon[29863]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:00.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:00 smithi186 ceph-mon[33438]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:01.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:00 smithi012 ceph-mon[29863]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:01.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:00 smithi012 ceph-mon[41141]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:02 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:02 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:02 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:02 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:02 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:02 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:02.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:02 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:02.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:02 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:02.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:02 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:03.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:03 smithi012 ceph-mon[29863]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:03.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:03 smithi012 ceph-mon[41141]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:03.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:03 smithi186 ceph-mon[33438]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:04.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:04 smithi186 ceph-mon[33438]: from='osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:45:04.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:04 smithi186 ceph-mon[33438]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:45:04.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:04.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:04 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:04.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[29863]: from='osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:45:04.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[29863]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:45:04.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:04.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:04.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[41141]: from='osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:45:04.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[41141]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:45:04.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:04.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:04 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.112+0000 7f806effd700 1 -- 172.21.15.186:0/3761983346 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f80780624c0 con 0x7f806406d9d0 2024-03-26T04:45:05.112 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 7 on host 'smithi186' 2024-03-26T04:45:05.114 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f806406d9d0 msgr2=0x7f806406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:05.114 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f806406d9d0 0x7f806406fe90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8074005950 tx=0x7f8074005f50 comp rx=0 tx=0).stop 2024-03-26T04:45:05.114 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 msgr2=0x7f807819e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f807819e100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f80680098e0 tx=0x7f806800f740 comp rx=0 tx=0).stop 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 shutdown_connections 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f806406d9d0 0x7f806406fe90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8078100930 0x7f807819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8078105c70 0x7f80781a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 --2- 172.21.15.186:0/3761983346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80781032a0 0x7f807819e100 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 >> 172.21.15.186:0/3761983346 conn(0x7f80780fc570 msgr2=0x7f8078101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 shutdown_connections 2024-03-26T04:45:05.115 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:05.114+0000 7f80801d4700 1 -- 172.21.15.186:0/3761983346 wait complete. 2024-03-26T04:45:05.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:45:05.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:45:05.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:45:05.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:05.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[29863]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:05 smithi012 ceph-mon[41141]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:45:05.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:45:05.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:45:05.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:45:05.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-26T04:45:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:05 smithi186 ceph-mon[33438]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-26T04:45:05.826 DEBUG:teuthology.orchestra.run.smithi186:osd.7> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T04:45:05.828 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T04:45:05.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:45:06.082 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:06.508 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:45:06.508 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.508 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.508 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: from='osd.7 ' entity='osd.7' 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] boot 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: from='osd.7 ' entity='osd.7' 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] boot 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T04:45:06.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:06 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: from='osd.7 ' entity='osd.7' 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: osd.7 [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] boot 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T04:45:06.688 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:06 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:45:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[29863]: purged_snaps scrub starts 2024-03-26T04:45:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[29863]: purged_snaps scrub ok 2024-03-26T04:45:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[29863]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[29863]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:45:07.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[41141]: purged_snaps scrub starts 2024-03-26T04:45:07.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[41141]: purged_snaps scrub ok 2024-03-26T04:45:07.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[41141]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:07.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:07 smithi012 ceph-mon[41141]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:45:07.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/2000914049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48107e40 msgr2=0x7f3d4810a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:07.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/2000914049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48107e40 0x7f3d4810a2b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009f90 tx=0x7f3d4000b330 comp rx=0 tx=0).stop 2024-03-26T04:45:07.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/2000914049 shutdown_connections 2024-03-26T04:45:07.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/2000914049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/2000914049 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48107e40 0x7f3d4810a2b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/2000914049 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48105510 0x7f3d48107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/2000914049 >> 172.21.15.12:0/2000914049 conn(0x7f3d480fc610 msgr2=0x7f3d480fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:07.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/2000914049 shutdown_connections 2024-03-26T04:45:07.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.708+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/2000914049 wait complete. 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.709+0000 7f3d4dfe4700 1 Processor -- start 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.709+0000 7f3d4dfe4700 1 -- start start 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d4dfe4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d4dfe4700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48105510 0x7f3d48100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d4dfe4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48107e40 0x7f3d481010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d4dfe4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48101850 con 0x7f3d48102be0 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d4dfe4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481019c0 con 0x7f3d48107e40 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d4dfe4700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481a1740 con 0x7f3d48105510 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d3effd700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48105510 0x7f3d48100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41290/0 (socket says 172.21.15.12:41290) 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 -- 172.21.15.12:0/415364656 learned_addr learned my addr 172.21.15.12:0/415364656 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:45:07.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d47fff700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48107e40 0x7f3d481010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 -- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48105510 msgr2=0x7f3d48100b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48105510 0x7f3d48100b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 -- 172.21.15.12:0/415364656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48107e40 msgr2=0x7f3d481010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48107e40 0x7f3d481010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 -- 172.21.15.12:0/415364656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380095f0 con 0x7f3d48102be0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d3effd700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48105510 0x7f3d48100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d47fff700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48107e40 0x7f3d481010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.710+0000 7f3d477fe700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48102540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800ec10 tx=0x7f3d3800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.711+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3800cc90 con 0x7f3d48102be0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.711+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3d3800cdf0 con 0x7f3d48102be0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.711+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d40009c40 con 0x7f3d48102be0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.711+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38003df0 con 0x7f3d48102be0 2024-03-26T04:45:07.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.711+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d481a1de0 con 0x7f3d48102be0 2024-03-26T04:45:07.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.712+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d480fe190 con 0x7f3d48102be0 2024-03-26T04:45:07.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.712+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f3d38021440 con 0x7f3d48102be0 2024-03-26T04:45:07.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.712+0000 7f3d457fa700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f3d3406d9d0 0x7f3d3406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:07.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.713+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5316+0+0 (secure 0 0 0) 0x7f3d38014070 con 0x7f3d48102be0 2024-03-26T04:45:07.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.713+0000 7f3d3effd700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f3d3406d9d0 0x7f3d3406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:07.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.713+0000 7f3d3effd700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f3d3406d9d0 0x7f3d3406fe90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d30011490 tx=0x7f3d30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:07.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:07 smithi186 ceph-mon[33438]: purged_snaps scrub starts 2024-03-26T04:45:07.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:07 smithi186 ceph-mon[33438]: purged_snaps scrub ok 2024-03-26T04:45:07.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:07 smithi186 ceph-mon[33438]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:45:07.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:07 smithi186 ceph-mon[33438]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:45:07.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.718+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3d380571b0 con 0x7f3d48102be0 2024-03-26T04:45:07.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.920+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3d4810cef0 con 0x7f3d48102be0 2024-03-26T04:45:07.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.920+0000 7f3d457fa700 1 -- 172.21.15.12:0/415364656 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v55) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3d3805a870 con 0x7f3d48102be0 2024-03-26T04:45:07.920 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.922+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f3d3406d9d0 msgr2=0x7f3d3406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.922+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f3d3406d9d0 0x7f3d3406fe90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d30011490 tx=0x7f3d30009260 comp rx=0 tx=0).stop 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.922+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 msgr2=0x7f3d48102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.922+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48102540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800ec10 tx=0x7f3d3800ef20 comp rx=0 tx=0).stop 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 shutdown_connections 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f3d3406d9d0 0x7f3d3406fe90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d48102be0 0x7f3d48102540 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d48107e40 0x7f3d481010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 --2- 172.21.15.12:0/415364656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f3d48105510 0x7f3d48100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 >> 172.21.15.12:0/415364656 conn(0x7f3d480fc610 msgr2=0x7f3d48103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 shutdown_connections 2024-03-26T04:45:07.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:07.923+0000 7f3d4dfe4700 1 -- 172.21.15.12:0/415364656 wait complete. 2024-03-26T04:45:08.331 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/415364656' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:45:08.331 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:08 smithi186 ceph-mon[33438]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:45:08.550 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711428306,"num_in_osds":8,"osd_in_since":1711428288,"num_remapped_pgs":1} 2024-03-26T04:45:08.551 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:45:08.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/415364656' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:45:08.576 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:08 smithi012 ceph-mon[29863]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:45:08.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/415364656' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:45:08.576 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:08 smithi012 ceph-mon[41141]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:45:08.793 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:09.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:09.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:09.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[29863]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:09.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:09 smithi012 ceph-mon[41141]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:09.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:09.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:09.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:09.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:09 smithi186 ceph-mon[33438]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:45:10.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[29863]: Detected new or changed devices on smithi186 2024-03-26T04:45:10.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi186 to 4272M 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[29863]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[41141]: Detected new or changed devices on smithi186 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi186 to 4272M 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:10.422 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:10 smithi012 ceph-mon[41141]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:45:10.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 -- 172.21.15.12:0/2241097590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107090 msgr2=0x7f890006a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:10.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 --2- 172.21.15.12:0/2241097590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107090 0x7f890006a720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009990 tx=0x7f88fc009ca0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 -- 172.21.15.12:0/2241097590 shutdown_connections 2024-03-26T04:45:10.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 --2- 172.21.15.12:0/2241097590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107090 0x7f890006a720 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 --2- 172.21.15.12:0/2241097590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900104b00 0x7f8900104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.424 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 --2- 172.21.15.12:0/2241097590 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900107470 0x7f890006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 -- 172.21.15.12:0/2241097590 >> 172.21.15.12:0/2241097590 conn(0x7f8900077e40 msgr2=0x7f8900078250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 -- 172.21.15.12:0/2241097590 shutdown_connections 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.423+0000 7f89077eb700 1 -- 172.21.15.12:0/2241097590 wait complete. 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 Processor -- start 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 -- start start 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 0x7f890019e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900107090 0x7f890019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107470 0x7f89001a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001a3020 con 0x7f8900107470 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001a3190 con 0x7f8900107090 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f89077eb700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89001a3350 con 0x7f8900104b00 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 0x7f890019e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:10.425 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905d88700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107470 0x7f89001a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 0x7f890019e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56692/0 (socket says 172.21.15.12:56692) 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 -- 172.21.15.12:0/2209396775 learned_addr learned my addr 172.21.15.12:0/2209396775 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8904d86700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900107090 0x7f890019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 -- 172.21.15.12:0/2209396775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900107090 msgr2=0x7f890019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900107090 0x7f890019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 -- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107470 msgr2=0x7f89001a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107470 0x7f89001a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905587700 1 -- 172.21.15.12:0/2209396775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8009c40 con 0x7f8900104b00 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.424+0000 7f8905d88700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107470 0x7f89001a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.425+0000 7f8905587700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 0x7f890019e190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88fc005740 tx=0x7f88fc005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.425+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc01d070 con 0x7f8900104b00 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.425+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f88fc00bd60 con 0x7f8900104b00 2024-03-26T04:45:10.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.425+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc0095f0 con 0x7f8900104b00 2024-03-26T04:45:10.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.425+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc022d20 con 0x7f8900104b00 2024-03-26T04:45:10.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.425+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89001aacc0 con 0x7f8900104b00 2024-03-26T04:45:10.427 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.426+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8900068d40 con 0x7f8900104b00 2024-03-26T04:45:10.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.426+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f88fc022570 con 0x7f8900104b00 2024-03-26T04:45:10.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.427+0000 7f88f67fc700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f88ec06d9d0 0x7f88ec06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:10.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.427+0000 7f8904d86700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f88ec06d9d0 0x7f88ec06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:10.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.427+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f88fc08e370 con 0x7f8900104b00 2024-03-26T04:45:10.428 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.427+0000 7f8904d86700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f88ec06d9d0 0x7f88ec06fe90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8900078660 tx=0x7f88f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:10.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.432+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f88fc058030 con 0x7f8900104b00 2024-03-26T04:45:10.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.636+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8900117cc0 con 0x7f8900104b00 2024-03-26T04:45:10.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.637+0000 7f88f67fc700 1 -- 172.21.15.12:0/2209396775 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13564 (secure 0 0 0) 0x7f88fc08ce60 con 0x7f8900104b00 2024-03-26T04:45:10.636 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:45:10.637 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":58,"fsid":"c3b00422-eb2a-11ee-95d0-87774f69a715","created":"2024-03-26T04:39:44.325097+0000","modified":"2024-03-26T04:45:10.287061+0000","last_up_change":"2024-03-26T04:45:06.281615+0000","last_in_change":"2024-03-26T04:44:48.805023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:43:00.923769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff69b85-c494-4958-862f-62bc607e0807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6803","nonce":3811666428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6805","nonce":3811666428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6809","nonce":3811666428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6807","nonce":3811666428}]},"public_addr":"172.21.15.12:6803/3811666428","cluster_addr":"172.21.15.12:6805/3811666428","heartbeat_back_addr":"172.21.15.12:6809/3811666428","heartbeat_front_addr":"172.21.15.12:6807/3811666428","state":["exists","up"]},{"osd":1,"uuid":"0dc7629c-67f1-498c-a7c8-434c9f9695c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6811","nonce":4077554022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6813","nonce":4077554022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6817","nonce":4077554022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6815","nonce":4077554022}]},"public_addr":"172.21.15.12:6811/4077554022","cluster_addr":"172.21.15.12:6813/4077554022","heartbeat_back_addr":"172.21.15.12:6817/4077554022","heartbeat_front_addr":"172.21.15.12:6815/4077554022","state":["exists","up"]},{"osd":2,"uuid":"185f5e96-6db7-41a5-9384-384dae329880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6819","nonce":4166122647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6821","nonce":4166122647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6825","nonce":4166122647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6823","nonce":4166122647}]},"public_addr":"172.21.15.12:6819/4166122647","cluster_addr":"172.21.15.12:6821/4166122647","heartbeat_back_addr":"172.21.15.12:6825/4166122647","heartbeat_front_addr":"172.21.15.12:6823/4166122647","state":["exists","up"]},{"osd":3,"uuid":"ec8405e3-d836-443e-98d6-998c0ceb0a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6827","nonce":3848141010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6829","nonce":3848141010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6833","nonce":3848141010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6831","nonce":3848141010}]},"public_addr":"172.21.15.12:6827/3848141010","cluster_addr":"172.21.15.12:6829/3848141010","heartbeat_back_addr":"172.21.15.12:6833/3848141010","heartbeat_front_addr":"172.21.15.12:6831/3848141010","state":["exists","up"]},{"osd":4,"uuid":"745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6801","nonce":3595955502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6803","nonce":3595955502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6807","nonce":3595955502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6805","nonce":3595955502}]},"public_addr":"172.21.15.186:6801/3595955502","cluster_addr":"172.21.15.186:6803/3595955502","heartbeat_back_addr":"172.21.15.186:6807/3595955502","heartbeat_front_addr":"172.21.15.186:6805/3595955502","state":["exists","up"]},{"osd":5,"uuid":"1f918fca-3303-4d6c-a0d5-81d952478987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6809","nonce":223280961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6811","nonce":223280961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6815","nonce":223280961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6813","nonce":223280961}]},"public_addr":"172.21.15.186:6809/223280961","cluster_addr":"172.21.15.186:6811/223280961","heartbeat_back_addr":"172.21.15.186:6815/223280961","heartbeat_front_addr":"172.21.15.186:6813/223280961","state":["exists","up"]},{"osd":6,"uuid":"bdc95ff2-55b2-4401-b0d7-d8b58beb916d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6817","nonce":2797487459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6819","nonce":2797487459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6823","nonce":2797487459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6821","nonce":2797487459}]},"public_addr":"172.21.15.186:6817/2797487459","cluster_addr":"172.21.15.186:6819/2797487459","heartbeat_back_addr":"172.21.15.186:6823/2797487459","heartbeat_front_addr":"172.21.15.186:6821/2797487459","state":["exists","up"]},{"osd":7,"uuid":"04551815-3856-4382-85e7-bd7b39e62bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6825","nonce":1742409046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6827","nonce":1742409046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6831","nonce":1742409046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6829","nonce":1742409046}]},"public_addr":"172.21.15.186:6825/1742409046","cluster_addr":"172.21.15.186:6827/1742409046","heartbeat_back_addr":"172.21.15.186:6831/1742409046","heartbeat_front_addr":"172.21.15.186:6829/1742409046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:09.131249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:33.743929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:58.585033+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:43:23.412593+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:43:47.048739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:44:12.322403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:44:38.578143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:45:05.219851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/284140021":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/1561816911":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/239771954":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1875274271":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1841124439":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1796197499":"2024-03-27T04:40:44.587143+0000","172.21.15.12:6800/2152405375":"2024-03-27T04:40:17.651952+0000","172.21.15.12:6800/284140021":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/865472781":"2024-03-27T04:40:44.587143+0000","172.21.15.12:6801/2152405375":"2024-03-27T04:40:17.651952+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:45:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f88ec06d9d0 msgr2=0x7f88ec06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:10.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f88ec06d9d0 0x7f88ec06fe90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8900078660 tx=0x7f88f0009260 comp rx=0 tx=0).stop 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 msgr2=0x7f890019e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 0x7f890019e190 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f88fc005740 tx=0x7f88fc005770 comp rx=0 tx=0).stop 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 shutdown_connections 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f88ec06d9d0 0x7f88ec06fe90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8900107470 0x7f89001a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8900104b00 0x7f890019e190 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 --2- 172.21.15.12:0/2209396775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8900107090 0x7f890019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:10.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 >> 172.21.15.12:0/2209396775 conn(0x7f8900077e40 msgr2=0x7f8900103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:10.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 shutdown_connections 2024-03-26T04:45:10.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:10.639+0000 7f89077eb700 1 -- 172.21.15.12:0/2209396775 wait complete. 2024-03-26T04:45:10.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:10 smithi186 ceph-mon[33438]: Detected new or changed devices on smithi186 2024-03-26T04:45:10.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:10 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi186 to 4272M 2024-03-26T04:45:10.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:10 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:10.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:10 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:10.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:10 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:10.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:10 smithi186 ceph-mon[33438]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:45:11.384 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:43:00.923769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T04:45:11.384 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T04:45:11.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:11 smithi012 ceph-mon[41141]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:11.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2209396775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:45:11.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:11 smithi012 ceph-mon[29863]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:11.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2209396775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:45:11.669 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:11.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:11 smithi186 ceph-mon[33438]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:11.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2209396775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:45:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:12 smithi186 ceph-mon[33438]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-26T04:45:13.000 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:12 smithi012 ceph-mon[29863]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-26T04:45:13.001 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:12 smithi012 ceph-mon[41141]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 -- 172.21.15.12:0/1986760515 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee01019e0 msgr2=0x7fdee010df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 --2- 172.21.15.12:0/1986760515 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee01019e0 0x7fdee010df50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fded8009f90 tx=0x7fded800b330 comp rx=0 tx=0).stop 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 -- 172.21.15.12:0/1986760515 shutdown_connections 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 --2- 172.21.15.12:0/1986760515 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee01019e0 0x7fdee010df50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 --2- 172.21.15.12:0/1986760515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee0102f80 0x7fdee0103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 --2- 172.21.15.12:0/1986760515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee0101020 0x7fdee01014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.392+0000 7fdee493c700 1 -- 172.21.15.12:0/1986760515 >> 172.21.15.12:0/1986760515 conn(0x7fdee0077020 msgr2=0x7fdee0077430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 -- 172.21.15.12:0/1986760515 shutdown_connections 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 -- 172.21.15.12:0/1986760515 wait complete. 2024-03-26T04:45:13.396 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 Processor -- start 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 -- start start 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 0x7fdee0103db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee01019e0 0x7fdee01042f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee0102f80 0x7fdee0104830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01a6060 con 0x7fdee01019e0 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01a61d0 con 0x7fdee0102f80 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.393+0000 7fdee493c700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee01a6390 con 0x7fdee0101020 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 0x7fdee0103db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdeddd9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee01019e0 0x7fdee01042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 0x7fdee0103db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56712/0 (socket says 172.21.15.12:56712) 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 -- 172.21.15.12:0/3328029501 learned_addr learned my addr 172.21.15.12:0/3328029501 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdeded9d700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee0102f80 0x7fdee0104830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 -- 172.21.15.12:0/3328029501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee0102f80 msgr2=0x7fdee0104830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:13.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee0102f80 0x7fdee0104830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 -- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee01019e0 msgr2=0x7fdee01042f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee01019e0 0x7fdee01042f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 -- 172.21.15.12:0/3328029501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00095f0 con 0x7fdee0101020 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdeddd9b700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee01019e0 0x7fdee01042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdede59c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 0x7fdee0103db0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fded400c9b0 tx=0x7fded400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4007b00 con 0x7fdee0101020 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fded400ce50 con 0x7fdee0101020 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded8009c40 con 0x7fdee0101020 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4018400 con 0x7fdee0101020 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.394+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee01a69e0 con 0x7fdee0101020 2024-03-26T04:45:13.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.395+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee0068d40 con 0x7fdee0101020 2024-03-26T04:45:13.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.396+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7fded400f450 con 0x7fdee0101020 2024-03-26T04:45:13.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.396+0000 7fdecf7fe700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fdec806d9d0 0x7fdec806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:13.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.396+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fded408ce00 con 0x7fdee0101020 2024-03-26T04:45:13.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.396+0000 7fdeddd9b700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fdec806d9d0 0x7fdec806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:13.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.397+0000 7fdeddd9b700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fdec806d9d0 0x7fdec806fe90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdee0111140 tx=0x7fded0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:13.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.401+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fded4056ac0 con 0x7fdee0101020 2024-03-26T04:45:13.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.603+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdee01105f0 con 0x7fdee0101020 2024-03-26T04:45:13.604 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-03-26T04:45:13.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.604+0000 7fdecf7fe700 1 -- 172.21.15.12:0/3328029501 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fded4007e60 con 0x7fdee0101020 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fdec806d9d0 msgr2=0x7fdec806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fdec806d9d0 0x7fdec806fe90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdee0111140 tx=0x7fded0009310 comp rx=0 tx=0).stop 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 msgr2=0x7fdee0103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 0x7fdee0103db0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fded400c9b0 tx=0x7fded400ccc0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 shutdown_connections 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fdec806d9d0 0x7fdec806fe90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdee0102f80 0x7fdee0104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdee0101020 0x7fdee0103db0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 --2- 172.21.15.12:0/3328029501 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee01019e0 0x7fdee01042f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:13.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.606+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 >> 172.21.15.12:0/3328029501 conn(0x7fdee0077020 msgr2=0x7fdee00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:13.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.607+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 shutdown_connections 2024-03-26T04:45:13.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:13.607+0000 7fdee493c700 1 -- 172.21.15.12:0/3328029501 wait complete. 2024-03-26T04:45:13.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3328029501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:45:13.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3328029501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:45:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3328029501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:45:14.255 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi012 2024-03-26T04:45:14.256 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi012=foo.a' 2024-03-26T04:45:14.507 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:14.898 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:14 smithi186 ceph-mon[33438]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T04:45:15.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:14 smithi012 ceph-mon[29863]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T04:45:15.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:14 smithi012 ceph-mon[41141]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T04:45:16.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 -- 172.21.15.186:0/1533910155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff0102b50 msgr2=0x7f2ff0109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:16.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 --2- 172.21.15.186:0/1533910155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff0102b50 0x7f2ff0109400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800b0d0 tx=0x7f2fe800b3e0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 -- 172.21.15.186:0/1533910155 shutdown_connections 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 --2- 172.21.15.186:0/1533910155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff0100fb0 0x7f2ff01013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 --2- 172.21.15.186:0/1533910155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff0102b50 0x7f2ff0109400 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 --2- 172.21.15.186:0/1533910155 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ff01021b0 0x7f2ff0102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 -- 172.21.15.186:0/1533910155 >> 172.21.15.186:0/1533910155 conn(0x7f2ff00fc570 msgr2=0x7f2ff00fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 -- 172.21.15.186:0/1533910155 shutdown_connections 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.113+0000 7f2ff524f700 1 -- 172.21.15.186:0/1533910155 wait complete. 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 Processor -- start 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 -- start start 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ff0100fb0 0x7f2ff010b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 0x7f2ff0109f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:16.117 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff0102b50 0x7f2ff010a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff010abd0 con 0x7f2ff0102b50 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff010ad40 con 0x7f2ff01021b0 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2ff524f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff01a16c0 con 0x7f2ff0100fb0 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2fee59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 0x7f2ff0109f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2fee59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 0x7f2ff0109f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36838/0 (socket says 172.21.15.186:36838) 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2fee59c700 1 -- 172.21.15.186:0/137737831 learned_addr learned my addr 172.21.15.186:0/137737831 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2feed9d700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ff0100fb0 0x7f2ff010b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.114+0000 7f2fef59e700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff0102b50 0x7f2ff010a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fee59c700 1 -- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ff0100fb0 msgr2=0x7f2ff010b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fee59c700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ff0100fb0 0x7f2ff010b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fee59c700 1 -- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff0102b50 msgr2=0x7f2ff010a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fee59c700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff0102b50 0x7f2ff010a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.118 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fee59c700 1 -- 172.21.15.186:0/137737831 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8009d00 con 0x7f2ff01021b0 2024-03-26T04:45:16.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fee59c700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 0x7f2ff0109f10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c920 tx=0x7f2fe000cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:16.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0004a30 con 0x7f2ff01021b0 2024-03-26T04:45:16.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2fe00054a0 con 0x7f2ff01021b0 2024-03-26T04:45:16.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff01a19a0 con 0x7f2ff01021b0 2024-03-26T04:45:16.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0007ab0 con 0x7f2ff01021b0 2024-03-26T04:45:16.119 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.115+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff01a1ea0 con 0x7f2ff01021b0 2024-03-26T04:45:16.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.116+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff0108370 con 0x7f2ff01021b0 2024-03-26T04:45:16.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.117+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f2fe0007c10 con 0x7f2ff01021b0 2024-03-26T04:45:16.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.117+0000 7f2fd7fff700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f2fd806d9d0 0x7f2fd806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:16.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.117+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f2fe008cd30 con 0x7f2ff01021b0 2024-03-26T04:45:16.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.117+0000 7f2feed9d700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f2fd806d9d0 0x7f2fd806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:16.120 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.118+0000 7f2feed9d700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f2fd806d9d0 0x7f2fd806fe90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0102010 tx=0x7f2fe400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:16.122 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.122+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2fe00569f0 con 0x7f2ff01021b0 2024-03-26T04:45:16.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.327+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi012=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ff00624c0 con 0x7f2fd806d9d0 2024-03-26T04:45:16.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.334+0000 7f2fd7fff700 1 -- 172.21.15.186:0/137737831 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f2ff00624c0 con 0x7f2fd806d9d0 2024-03-26T04:45:16.335 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled rgw.foo.a update... 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f2fd806d9d0 msgr2=0x7f2fd806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f2fd806d9d0 0x7f2fd806fe90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0102010 tx=0x7f2fe400a380 comp rx=0 tx=0).stop 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 msgr2=0x7f2ff0109f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 0x7f2ff0109f10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c920 tx=0x7f2fe000cc30 comp rx=0 tx=0).stop 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 shutdown_connections 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f2fd806d9d0 0x7f2fd806fe90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ff0102b50 0x7f2ff010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2ff0100fb0 0x7f2ff010b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 --2- 172.21.15.186:0/137737831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff01021b0 0x7f2ff0109f10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:16.337 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 >> 172.21.15.186:0/137737831 conn(0x7f2ff00fc570 msgr2=0x7f2ff01054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:16.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 shutdown_connections 2024-03-26T04:45:16.338 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:16.336+0000 7f2ff524f700 1 -- 172.21.15.186:0/137737831 wait complete. 2024-03-26T04:45:16.954 DEBUG:teuthology.orchestra.run.smithi012:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T04:45:16.957 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi186 2024-03-26T04:45:16.957 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T04:45:17.196 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:17.594 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi012=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:17.594 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: Saving service rgw.foo.a spec with placement smithi012=foo.a;count:1 2024-03-26T04:45:17.594 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: Deploying daemon rgw.foo.a on smithi012 2024-03-26T04:45:17.595 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:17 smithi186 ceph-mon[33438]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi012=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: Saving service rgw.foo.a spec with placement smithi012=foo.a;count:1 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: Deploying daemon rgw.foo.a on smithi012 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[41141]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi012=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: Saving service rgw.foo.a spec with placement smithi012=foo.a;count:1 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:45:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:45:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: Deploying daemon rgw.foo.a on smithi012 2024-03-26T04:45:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:17 smithi012 ceph-mon[29863]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:45:18.642 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 systemd[1]: Starting Ceph rgw.foo.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.891+0000 7f0aace43700 1 -- 172.21.15.186:0/1586072386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa8102b90 msgr2=0x7f0aa8109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.891+0000 7f0aace43700 1 --2- 172.21.15.186:0/1586072386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa8102b90 0x7f0aa8109440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000b0d0 tx=0x7f0aa000b3e0 comp rx=0 tx=0).stop 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 -- 172.21.15.186:0/1586072386 shutdown_connections 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 --2- 172.21.15.186:0/1586072386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa8100ff0 0x7f0aa8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 --2- 172.21.15.186:0/1586072386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa8102b90 0x7f0aa8109440 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 --2- 172.21.15.186:0/1586072386 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa81021f0 0x7f0aa8102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 -- 172.21.15.186:0/1586072386 >> 172.21.15.186:0/1586072386 conn(0x7f0aa80fc570 msgr2=0x7f0aa80fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 -- 172.21.15.186:0/1586072386 shutdown_connections 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 -- 172.21.15.186:0/1586072386 wait complete. 2024-03-26T04:45:18.895 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 Processor -- start 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.892+0000 7f0aace43700 1 -- start start 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 0x7f0aa806b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 0x7f0aa819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa8102b90 0x7f0aa819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa819f1d0 con 0x7f0aa8102b90 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa819f340 con 0x7f0aa81021f0 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa819f500 con 0x7f0aa8100ff0 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 0x7f0aa819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 0x7f0aa806b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 0x7f0aa806b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.186:38194/0 (socket says 172.21.15.186:38194) 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 0x7f0aa819c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36862/0 (socket says 172.21.15.186:36862) 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 -- 172.21.15.186:0/4158637656 learned_addr learned my addr 172.21.15.186:0/4158637656 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa6d9d700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa8102b90 0x7f0aa819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 -- 172.21.15.186:0/4158637656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 msgr2=0x7f0aa819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:18.896 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 0x7f0aa819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 -- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa8102b90 msgr2=0x7f0aa819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa8102b90 0x7f0aa819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 -- 172.21.15.186:0/4158637656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa0009d00 con 0x7f0aa8100ff0 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa5d9b700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 0x7f0aa819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aa659c700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 0x7f0aa806b2b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800d980 tx=0x7f0a9800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9800f890 con 0x7f0aa8100ff0 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa819f7e0 con 0x7f0aa8100ff0 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0a9800fe70 con 0x7f0aa8100ff0 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.893+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98010e80 con 0x7f0aa8100ff0 2024-03-26T04:45:18.897 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.894+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa819fce0 con 0x7f0aa8100ff0 2024-03-26T04:45:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.894+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa8107bd0 con 0x7f0aa8100ff0 2024-03-26T04:45:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.895+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f0a9800f9f0 con 0x7f0aa8100ff0 2024-03-26T04:45:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.895+0000 7f0a977fe700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0a9006d9d0 0x7f0a9006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.895+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f0a9808cc50 con 0x7f0aa8100ff0 2024-03-26T04:45:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.895+0000 7f0aa5d9b700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0a9006d9d0 0x7f0a9006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:18.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.896+0000 7f0aa5d9b700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0a9006d9d0 0x7f0a9006fe90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009fd0 tx=0x7f0a9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:18.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:18.900+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0a98092020 con 0x7f0aa8100ff0 2024-03-26T04:45:18.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:18 smithi012 ceph-mon[29863]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-26T04:45:18.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:18 smithi012 ceph-mon[41141]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-26T04:45:18.905 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 podman[68097]: 2024-03-26 04:45:18.642512915 +0000 UTC m=+0.107941162 container create f5bdc9f0bdbe7afbcbd475d3a430bdfcfcffbad5c50bb4fb4eb1d863b19f84cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:45:18.905 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 podman[68097]: 2024-03-26 04:45:18.553045382 +0000 UTC m=+0.018473641 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:45:18.906 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 podman[68097]: 2024-03-26 04:45:18.87047118 +0000 UTC m=+0.335899428 container init f5bdc9f0bdbe7afbcbd475d3a430bdfcfcffbad5c50bb4fb4eb1d863b19f84cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T04:45:18.906 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 podman[68097]: 2024-03-26 04:45:18.873295345 +0000 UTC m=+0.338723599 container start f5bdc9f0bdbe7afbcbd475d3a430bdfcfcffbad5c50bb4fb4eb1d863b19f84cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T04:45:18.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:18 smithi186 ceph-mon[33438]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-26T04:45:19.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.095+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f0aa810bf10 con 0x7f0aa8100ff0 2024-03-26T04:45:19.405 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 bash[68097]: f5bdc9f0bdbe7afbcbd475d3a430bdfcfcffbad5c50bb4fb4eb1d863b19f84cc 2024-03-26T04:45:19.405 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 04:45:18 smithi012 systemd[1]: Started Ceph rgw.foo.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:45:19.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.719+0000 7f0a977fe700 1 -- 172.21.15.186:0/4158637656 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (secure 0 0 0) 0x7f0a98056910 con 0x7f0aa8100ff0 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.723+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0a9006d9d0 msgr2=0x7f0a9006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.723+0000 7f0aace43700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0a9006d9d0 0x7f0a9006fe90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009fd0 tx=0x7f0a9c009260 comp rx=0 tx=0).stop 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.723+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 msgr2=0x7f0aa806b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 0x7f0aa806b2b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800d980 tx=0x7f0a9800dc90 comp rx=0 tx=0).stop 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 shutdown_connections 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0a9006d9d0 0x7f0a9006fe90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0aa8102b90 0x7f0aa819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0aa8100ff0 0x7f0aa806b2b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 --2- 172.21.15.186:0/4158637656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa81021f0 0x7f0aa819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 >> 172.21.15.186:0/4158637656 conn(0x7f0aa80fc570 msgr2=0x7f0aa80fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 shutdown_connections 2024-03-26T04:45:19.725 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:19.724+0000 7f0aace43700 1 -- 172.21.15.186:0/4158637656 wait complete. 2024-03-26T04:45:19.726 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'datapool' created 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/4158637656' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: Saving service rgw.foo.a spec with placement smithi012=foo.a;count:1 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:19.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:19 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/4158637656' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:45:20.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: Saving service rgw.foo.a spec with placement smithi012=foo.a;count:1 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/4158637656' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:45:20.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: Saving service rgw.foo.a spec with placement smithi012=foo.a;count:1 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:20.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:19 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:20.359 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T04:45:20.595 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:21.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T04:45:21.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:20 smithi186 ceph-mon[33438]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:45:21.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/36392835' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:45:21.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:45:21.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:20 smithi186 ceph-mon[33438]: pgmap v187: 36 pgs: 16 creating+peering, 1 active+clean, 19 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:45:21.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T04:45:21.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[29863]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:45:21.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/36392835' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:45:21.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:45:21.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[29863]: pgmap v187: 36 pgs: 16 creating+peering, 1 active+clean, 19 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:45:21.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T04:45:21.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[41141]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:45:21.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/36392835' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:45:21.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:45:21.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:20 smithi012 ceph-mon[41141]: pgmap v187: 36 pgs: 16 creating+peering, 1 active+clean, 19 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:45:21.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:21.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:21 smithi186 ceph-mon[33438]: Checking dashboard <-> RGW credentials 2024-03-26T04:45:22.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:45:22.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:45:22.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:45:22.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[29863]: Checking dashboard <-> RGW credentials 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:21 smithi012 ceph-mon[41141]: Checking dashboard <-> RGW credentials 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/2077999341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:45:23.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[29863]: pgmap v190: 68 pgs: 21 creating+peering, 14 active+clean, 33 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/2077999341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:45:23.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:22 smithi012 ceph-mon[41141]: pgmap v190: 68 pgs: 21 creating+peering, 14 active+clean, 33 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/2077999341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:45:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:22 smithi186 ceph-mon[33438]: pgmap v190: 68 pgs: 21 creating+peering, 14 active+clean, 33 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[29863]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:45:24.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T04:45:24.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:23 smithi012 ceph-mon[41141]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:45:24.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:45:24.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:23 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:45:24.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T04:45:24.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:23 smithi186 ceph-mon[33438]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:45:25.134 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:24 smithi186 ceph-mon[33438]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:45:25.135 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.135 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:24 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.135 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.135 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:24 smithi186 ceph-mon[33438]: pgmap v193: 100 pgs: 25 creating+peering, 24 active+clean, 51 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[29863]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[29863]: pgmap v193: 100 pgs: 25 creating+peering, 24 active+clean, 51 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[41141]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:45:25.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:24 smithi012 ceph-mon[41141]: pgmap v193: 100 pgs: 25 creating+peering, 24 active+clean, 51 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T04:45:25.452 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.186 --placement '1;smithi186=iscsi.a' 2024-03-26T04:45:25.696 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:26.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:25 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:45:26.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:45:26.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:25 smithi186 ceph-mon[33438]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:45:26.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:25 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:45:26.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:45:26.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:25 smithi012 ceph-mon[29863]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:45:26.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:25 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:45:26.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:45:26.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:25 smithi012 ceph-mon[41141]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:45:27.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:26 smithi186 ceph-mon[33438]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:45:27.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:26 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.103 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:26 smithi186 ceph-mon[33438]: pgmap v196: 132 pgs: 24 creating+peering, 76 active+clean, 32 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T04:45:27.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[29863]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:45:27.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[29863]: pgmap v196: 132 pgs: 24 creating+peering, 76 active+clean, 32 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T04:45:27.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[41141]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:45:27.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:45:27.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:26 smithi012 ceph-mon[41141]: pgmap v196: 132 pgs: 24 creating+peering, 76 active+clean, 32 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T04:45:27.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 -- 172.21.15.186:0/1276850011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754101980 msgr2=0x7f2754101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:27.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 --2- 172.21.15.186:0/1276850011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754101980 0x7f2754101de0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f27440099c0 tx=0x7f2744009cd0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 -- 172.21.15.186:0/1276850011 shutdown_connections 2024-03-26T04:45:27.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 --2- 172.21.15.186:0/1276850011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 0x7f2754100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 --2- 172.21.15.186:0/1276850011 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2754102320 0x7f2754108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 --2- 172.21.15.186:0/1276850011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754101980 0x7f2754101de0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 -- 172.21.15.186:0/1276850011 >> 172.21.15.186:0/1276850011 conn(0x7f27540fbd20 msgr2=0x7f27540fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:27.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 -- 172.21.15.186:0/1276850011 shutdown_connections 2024-03-26T04:45:27.357 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.355+0000 7f275bd2b700 1 -- 172.21.15.186:0/1276850011 wait complete. 2024-03-26T04:45:27.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 Processor -- start 2024-03-26T04:45:27.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 -- start start 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 0x7f2754197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2754101980 0x7f2754197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754102320 0x7f275419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275419efd0 con 0x7f2754100780 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275419f110 con 0x7f2754102320 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275bd2b700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275419f2a0 con 0x7f2754101980 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f2759ac7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 0x7f2754197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275a2c8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754102320 0x7f275419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f2759ac7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 0x7f2754197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.186:33034/0 (socket says 172.21.15.186:33034) 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f2759ac7700 1 -- 172.21.15.186:0/2482359721 learned_addr learned my addr 172.21.15.186:0/2482359721 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f27592c6700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2754101980 0x7f2754197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275a2c8700 1 -- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2754101980 msgr2=0x7f2754197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:27.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275a2c8700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2754101980 0x7f2754197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275a2c8700 1 -- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 msgr2=0x7f2754197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275a2c8700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 0x7f2754197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.356+0000 7f275a2c8700 1 -- 172.21.15.186:0/2482359721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27440095f0 con 0x7f2754102320 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.357+0000 7f275a2c8700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754102320 0x7f275419ea00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f274c00d170 tx=0x7f274c00d480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.357+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c017070 con 0x7f2754102320 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.357+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f274c00dd10 con 0x7f2754102320 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.357+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c00f040 con 0x7f2754102320 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.357+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2754102ba0 con 0x7f2754102320 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.357+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2754103050 con 0x7f2754102320 2024-03-26T04:45:27.360 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.358+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2754050590 con 0x7f2754102320 2024-03-26T04:45:27.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.359+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f274c00a0d0 con 0x7f2754102320 2024-03-26T04:45:27.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.359+0000 7f274affd700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f274006da70 0x7f274006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:27.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.359+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7132+0+0 (secure 0 0 0) 0x7f274c08cf70 con 0x7f2754102320 2024-03-26T04:45:27.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.359+0000 7f2759ac7700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f274006da70 0x7f274006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:27.361 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.360+0000 7f2759ac7700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f274006da70 0x7f274006ff30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f275000ace0 tx=0x7f275000a430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:27.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.364+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f274c0c1280 con 0x7f2754102320 2024-03-26T04:45:27.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.571+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.186", "placement": "1;smithi186=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f27541085b0 con 0x7f274006da70 2024-03-26T04:45:27.580 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.579+0000 7f274affd700 1 -- 172.21.15.186:0/2482359721 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f27541085b0 con 0x7f274006da70 2024-03-26T04:45:27.580 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled iscsi.datapool update... 2024-03-26T04:45:27.582 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f274006da70 msgr2=0x7f274006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f274006da70 0x7f274006ff30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f275000ace0 tx=0x7f275000a430 comp rx=0 tx=0).stop 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754102320 msgr2=0x7f275419ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754102320 0x7f275419ea00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f274c00d170 tx=0x7f274c00d480 comp rx=0 tx=0).stop 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 shutdown_connections 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f274006da70 0x7f274006ff30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2754100780 0x7f2754197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2754102320 0x7f275419ea00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 --2- 172.21.15.186:0/2482359721 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2754101980 0x7f2754197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.581+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 >> 172.21.15.186:0/2482359721 conn(0x7f27540fbd20 msgr2=0x7f2754106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.582+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 shutdown_connections 2024-03-26T04:45:27.583 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:27.582+0000 7f275bd2b700 1 -- 172.21.15.186:0/2482359721 wait complete. 2024-03-26T04:45:28.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:45:28.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:45:28.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:45:28.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.186", "placement": "1;smithi186=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: Saving service iscsi.datapool spec with placement smithi186=iscsi.a;count:1 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.186", "placement": "1;smithi186=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: Saving service iscsi.datapool spec with placement smithi186=iscsi.a;count:1 2024-03-26T04:45:28.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:27 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:28.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3848070558' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.186", "placement": "1;smithi186=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: Saving service iscsi.datapool spec with placement smithi186=iscsi.a;count:1 2024-03-26T04:45:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:27 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:28.336 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T04:45:28.336 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:45:28.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T04:45:28.365 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:45:28.365 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T04:45:28.396 DEBUG:teuthology.orchestra.run.smithi186:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T04:45:28.439 INFO:tasks.cephadm:Adding prometheus.a on smithi186 2024-03-26T04:45:28.439 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi186=a' 2024-03-26T04:45:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: Deploying daemon iscsi.iscsi.a on smithi186 2024-03-26T04:45:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:28 smithi186 ceph-mon[33438]: pgmap v199: 132 pgs: 24 creating+peering, 94 active+clean, 14 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 1 op/s 2024-03-26T04:45:29.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: Deploying daemon iscsi.iscsi.a on smithi186 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[41141]: pgmap v199: 132 pgs: 24 creating+peering, 94 active+clean, 14 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 1 op/s 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:45:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4280417874' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: Deploying daemon iscsi.iscsi.a on smithi186 2024-03-26T04:45:29.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:28 smithi012 ceph-mon[29863]: pgmap v199: 132 pgs: 24 creating+peering, 94 active+clean, 14 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 1 op/s 2024-03-26T04:45:29.765 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:29.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:29 smithi186 ceph-mon[33438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:45:29.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:29 smithi186 ceph-mon[33438]: Cluster is now healthy 2024-03-26T04:45:29.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:29 smithi186 ceph-mon[33438]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:45:30.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:29 smithi012 ceph-mon[29863]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:45:30.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:29 smithi012 ceph-mon[29863]: Cluster is now healthy 2024-03-26T04:45:30.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:29 smithi012 ceph-mon[29863]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:45:30.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:29 smithi012 ceph-mon[41141]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:45:30.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:29 smithi012 ceph-mon[41141]: Cluster is now healthy 2024-03-26T04:45:30.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:29 smithi012 ceph-mon[41141]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:45:30.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:30 smithi186 systemd[1]: Starting Ceph iscsi.iscsi.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:45:31.070 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:30 smithi186 ceph-mon[33438]: pgmap v201: 132 pgs: 12 creating+peering, 120 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 4.9 KiB/s wr, 198 op/s 2024-03-26T04:45:31.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:30 smithi012 ceph-mon[29863]: pgmap v201: 132 pgs: 12 creating+peering, 120 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 4.9 KiB/s wr, 198 op/s 2024-03-26T04:45:31.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:30 smithi012 ceph-mon[41141]: pgmap v201: 132 pgs: 12 creating+peering, 120 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 4.9 KiB/s wr, 198 op/s 2024-03-26T04:45:31.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 podman[60609]: 2024-03-26 04:45:30.974876116 +0000 UTC m=+0.020185917 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:45:31.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 podman[60609]: 2024-03-26 04:45:31.073711429 +0000 UTC m=+0.119021214 container create 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:45:31.713 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 podman[60609]: 2024-03-26 04:45:31.544869855 +0000 UTC m=+0.590179641 container init 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:45:31.714 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 podman[60609]: 2024-03-26 04:45:31.548195909 +0000 UTC m=+0.593505695 container start 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 -- 172.21.15.186:0/2829747148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f4103c80 msgr2=0x7f70f410a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 --2- 172.21.15.186:0/2829747148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f4103c80 0x7f70f410a530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70ec00b0d0 tx=0x7f70ec00b3e0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 -- 172.21.15.186:0/2829747148 shutdown_connections 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 --2- 172.21.15.186:0/2829747148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f4103c80 0x7f70f410a530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 --2- 172.21.15.186:0/2829747148 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70f41020e0 0x7f70f4102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 --2- 172.21.15.186:0/2829747148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70f41032e0 0x7f70f4103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 -- 172.21.15.186:0/2829747148 >> 172.21.15.186:0/2829747148 conn(0x7f70f40fd680 msgr2=0x7f70f40ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:31.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 -- 172.21.15.186:0/2829747148 shutdown_connections 2024-03-26T04:45:31.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 -- 172.21.15.186:0/2829747148 wait complete. 2024-03-26T04:45:31.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 Processor -- start 2024-03-26T04:45:31.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.734+0000 7f70fa332700 1 -- start start 2024-03-26T04:45:31.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70fa332700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 0x7f70f4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:31.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70fa332700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70f41032e0 0x7f70f4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:31.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70fa332700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70f4103c80 0x7f70f419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:31.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70fa332700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f419f1f0 con 0x7f70f41032e0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70fa332700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f419f330 con 0x7f70f41020e0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70fa332700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f419f4c0 con 0x7f70f4103c80 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 0x7f70f4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 0x7f70f4197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:42368/0 (socket says 172.21.15.186:42368) 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 -- 172.21.15.186:0/345608604 learned_addr learned my addr 172.21.15.186:0/345608604 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f88cf700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70f4103c80 0x7f70f419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f37fe700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70f41032e0 0x7f70f4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 -- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70f4103c80 msgr2=0x7f70f419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70f4103c80 0x7f70f419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 -- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70f41032e0 msgr2=0x7f70f4197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70f41032e0 0x7f70f4197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 -- 172.21.15.186:0/345608604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec009d00 con 0x7f70f41020e0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.735+0000 7f70f3fff700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 0x7f70f4197660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70e400d980 tx=0x7f70e400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.736+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4004020 con 0x7f70f41020e0 2024-03-26T04:45:31.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.736+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f70e400f690 con 0x7f70f41020e0 2024-03-26T04:45:31.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.736+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f419f7a0 con 0x7f70f41020e0 2024-03-26T04:45:31.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.736+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4003980 con 0x7f70f41020e0 2024-03-26T04:45:31.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.736+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f419fca0 con 0x7f70f41020e0 2024-03-26T04:45:31.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.736+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f41094a0 con 0x7f70f41020e0 2024-03-26T04:45:31.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.737+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) v1 ==== 91398+0+0 (secure 0 0 0) 0x7f70e4004180 con 0x7f70f41020e0 2024-03-26T04:45:31.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.738+0000 7f70f17fa700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f70dc06d9d0 0x7f70dc06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:31.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.738+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f70e408d1b0 con 0x7f70f41020e0 2024-03-26T04:45:31.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.738+0000 7f70f37fe700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f70dc06d9d0 0x7f70dc06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:31.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.738+0000 7f70f37fe700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f70dc06d9d0 0x7f70dc06fe90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f70e8005950 tx=0x7f70e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:31.743 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.743+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f70e4056730 con 0x7f70f41020e0 2024-03-26T04:45:31.982 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.980+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f70f40624c0 con 0x7f70dc06d9d0 2024-03-26T04:45:31.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug Started the configuration object watcher 2024-03-26T04:45:31.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug Checking for config object changes every 1s 2024-03-26T04:45:31.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug Processing osd blocklist entries for this node 2024-03-26T04:45:31.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 bash[60609]: 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 2024-03-26T04:45:31.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:31 smithi186 systemd[1]: Started Ceph iscsi.iscsi.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:45:31.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.988+0000 7f70f17fa700 1 -- 172.21.15.186:0/345608604 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f70f40624c0 con 0x7f70dc06d9d0 2024-03-26T04:45:31.988 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled prometheus update... 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f70dc06d9d0 msgr2=0x7f70dc06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f70dc06d9d0 0x7f70dc06fe90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f70e8005950 tx=0x7f70e800b410 comp rx=0 tx=0).stop 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 msgr2=0x7f70f4197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 0x7f70f4197660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70e400d980 tx=0x7f70e400dc90 comp rx=0 tx=0).stop 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 shutdown_connections 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f70dc06d9d0 0x7f70dc06fe90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f70f4103c80 0x7f70f419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f41020e0 0x7f70f4197660 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 --2- 172.21.15.186:0/345608604 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f70f41032e0 0x7f70f4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 >> 172.21.15.186:0/345608604 conn(0x7f70f40fd680 msgr2=0x7f70f41065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 shutdown_connections 2024-03-26T04:45:31.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:31.990+0000 7f70fa332700 1 -- 172.21.15.186:0/345608604 wait complete. 2024-03-26T04:45:32.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug Reading the configuration object to update local LIO configuration 2024-03-26T04:45:32.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug Configuration does not have an entry for this host(smithi186.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T04:45:32.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T04:45:33.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/2308738908' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:45:33.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:32 smithi186 ceph-mon[33438]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 5.8 KiB/s wr, 267 op/s 2024-03-26T04:45:33.162 DEBUG:teuthology.orchestra.run.smithi186:prometheus.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T04:45:33.164 INFO:tasks.cephadm:Adding node-exporter.a on smithi012 2024-03-26T04:45:33.164 INFO:tasks.cephadm:Adding node-exporter.b on smithi186 2024-03-26T04:45:33.164 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi012=a;smithi186=b' 2024-03-26T04:45:33.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/2308738908' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[29863]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 5.8 KiB/s wr, 267 op/s 2024-03-26T04:45:33.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/2308738908' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:45:33.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:32 smithi012 ceph-mon[41141]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 5.8 KiB/s wr, 267 op/s 2024-03-26T04:45:33.967 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:34.291 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:33 smithi186 ceph-mon[33438]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:45:34.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:33 smithi012 ceph-mon[29863]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:45:34.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:33 smithi012 ceph-mon[41141]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:45:35.396 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:35 smithi186 ceph-mon[33438]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 4.4 KiB/s wr, 203 op/s 2024-03-26T04:45:35.396 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:35 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:35.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:35 smithi012 ceph-mon[29863]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 4.4 KiB/s wr, 203 op/s 2024-03-26T04:45:35.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:35 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:35.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:35 smithi012 ceph-mon[41141]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 4.4 KiB/s wr, 203 op/s 2024-03-26T04:45:35.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:35 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:35.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 -- 172.21.15.186:0/4058976391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860102540 msgr2=0x7f0860102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 --2- 172.21.15.186:0/4058976391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860102540 0x7f0860102960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0850009990 tx=0x7f0850009ca0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 -- 172.21.15.186:0/4058976391 shutdown_connections 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 --2- 172.21.15.186:0/4058976391 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0860101c20 0x7f08601094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 --2- 172.21.15.186:0/4058976391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860102540 0x7f0860102960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 --2- 172.21.15.186:0/4058976391 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101280 0x7f08601016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 -- 172.21.15.186:0/4058976391 >> 172.21.15.186:0/4058976391 conn(0x7f08600fbcf0 msgr2=0x7f08600fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 -- 172.21.15.186:0/4058976391 shutdown_connections 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.732+0000 7f0867e9b700 1 -- 172.21.15.186:0/4058976391 wait complete. 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 Processor -- start 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 -- start start 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 0x7f0860197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:35.736 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101c20 0x7f0860197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0860102540 0x7f086019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086019f1f0 con 0x7f0860102540 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086019f330 con 0x7f0860101280 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0867e9b700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086019f4c0 con 0x7f0860101c20 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0865c37700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 0x7f0860197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0865c37700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 0x7f0860197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36092/0 (socket says 172.21.15.186:36092) 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0865c37700 1 -- 172.21.15.186:0/841662469 learned_addr learned my addr 172.21.15.186:0/841662469 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0866438700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0860102540 0x7f086019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.733+0000 7f0865436700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101c20 0x7f0860197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865c37700 1 -- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101c20 msgr2=0x7f0860197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865c37700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101c20 0x7f0860197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.737 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865c37700 1 -- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0860102540 msgr2=0x7f086019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865c37700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0860102540 0x7f086019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865c37700 1 -- 172.21.15.186:0/841662469 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500095f0 con 0x7f0860101280 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865436700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101c20 0x7f0860197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0865c37700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 0x7f0860197660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08500098e0 tx=0x7f085000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085001d070 con 0x7f0860101280 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0850022470 con 0x7f0860101280 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086019f740 con 0x7f0860101280 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850003a20 con 0x7f0860101280 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.734+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086019fc30 con 0x7f0860101280 2024-03-26T04:45:35.738 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.735+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0860107c70 con 0x7f0860101280 2024-03-26T04:45:35.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.735+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) v1 ==== 91449+0+0 (secure 0 0 0) 0x7f085000fac0 con 0x7f0860101280 2024-03-26T04:45:35.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.736+0000 7f0856ffd700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f084c06da60 0x7f084c06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:35.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.736+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f085008ed60 con 0x7f0860101280 2024-03-26T04:45:35.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.736+0000 7f0865436700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f084c06da60 0x7f084c06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:35.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.737+0000 7f0865436700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f084c06da60 0x7f084c06ff20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f085c0099d0 tx=0x7f085c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:35.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.741+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0850058260 con 0x7f0860101280 2024-03-26T04:45:35.948 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.947+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi012=a;smithi186=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f08600624c0 con 0x7f084c06da60 2024-03-26T04:45:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.955+0000 7f0856ffd700 1 -- 172.21.15.186:0/841662469 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f08600624c0 con 0x7f084c06da60 2024-03-26T04:45:35.956 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled node-exporter update... 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f084c06da60 msgr2=0x7f084c06ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f084c06da60 0x7f084c06ff20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f085c0099d0 tx=0x7f085c008040 comp rx=0 tx=0).stop 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 msgr2=0x7f0860197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 0x7f0860197660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08500098e0 tx=0x7f085000f690 comp rx=0 tx=0).stop 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 shutdown_connections 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f084c06da60 0x7f084c06ff20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0860102540 0x7f086019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0860101280 0x7f0860197660 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 --2- 172.21.15.186:0/841662469 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0860101c20 0x7f0860197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 >> 172.21.15.186:0/841662469 conn(0x7f08600fbcf0 msgr2=0x7f08600fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 shutdown_connections 2024-03-26T04:45:35.958 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:35.958+0000 7f0867e9b700 1 -- 172.21.15.186:0/841662469 wait complete. 2024-03-26T04:45:36.742 DEBUG:teuthology.orchestra.run.smithi012:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T04:45:36.745 DEBUG:teuthology.orchestra.run.smithi186:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T04:45:36.748 INFO:tasks.cephadm:Adding alertmanager.a on smithi012 2024-03-26T04:45:36.748 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi012=a' 2024-03-26T04:45:37.137 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[29863]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi012=a;smithi186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[29863]: Saving service node-exporter spec with placement smithi012=a;smithi186=b;count:2 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[29863]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 4.1 KiB/s wr, 182 op/s 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[41141]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi012=a;smithi186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[41141]: Saving service node-exporter spec with placement smithi012=a;smithi186=b;count:2 2024-03-26T04:45:37.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:37.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:36 smithi012 ceph-mon[41141]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 4.1 KiB/s wr, 182 op/s 2024-03-26T04:45:37.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:36 smithi186 ceph-mon[33438]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi012=a;smithi186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:37.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:36 smithi186 ceph-mon[33438]: Saving service node-exporter spec with placement smithi012=a;smithi186=b;count:2 2024-03-26T04:45:37.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:36 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:37.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:36 smithi186 ceph-mon[33438]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 4.1 KiB/s wr, 182 op/s 2024-03-26T04:45:38.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:38.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:38.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[41141]: Deploying daemon prometheus.a on smithi186 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:38 smithi012 ceph-mon[29863]: Deploying daemon prometheus.a on smithi186 2024-03-26T04:45:38.417 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.417 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.417 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:38.417 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:38.417 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:38 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:38.417 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:38 smithi186 ceph-mon[33438]: Deploying daemon prometheus.a on smithi186 2024-03-26T04:45:38.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 -- 172.21.15.186:0/512285424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1020d0 msgr2=0x7f1a5c1024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:38.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 --2- 172.21.15.186:0/512285424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1020d0 0x7f1a5c1024f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0099c0 tx=0x7f1a4c009cd0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 -- 172.21.15.186:0/512285424 shutdown_connections 2024-03-26T04:45:38.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 --2- 172.21.15.186:0/512285424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a5c103c70 0x7f1a5c10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.765 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 --2- 172.21.15.186:0/512285424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1020d0 0x7f1a5c1024f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 --2- 172.21.15.186:0/512285424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a5c1032d0 0x7f1a5c103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 -- 172.21.15.186:0/512285424 >> 172.21.15.186:0/512285424 conn(0x7f1a5c0fd690 msgr2=0x7f1a5c0ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 -- 172.21.15.186:0/512285424 shutdown_connections 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.760+0000 7f1a61692700 1 -- 172.21.15.186:0/512285424 wait complete. 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 Processor -- start 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 -- start start 2024-03-26T04:45:38.766 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a5c1020d0 0x7f1a5c075e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 0x7f1a5c074470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a5c103c70 0x7f1a5c0749b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c075130 con 0x7f1a5c1020d0 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0752a0 con 0x7f1a5c1032d0 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.761+0000 7f1a61692700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1a15f0 con 0x7f1a5c103c70 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 0x7f1a5c074470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 0x7f1a5c074470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36118/0 (socket says 172.21.15.186:36118) 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 -- 172.21.15.186:0/2890789083 learned_addr learned my addr 172.21.15.186:0/2890789083 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5affd700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a5c1020d0 0x7f1a5c075e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5b7fe700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a5c103c70 0x7f1a5c0749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:38.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 -- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a5c103c70 msgr2=0x7f1a5c0749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a5c103c70 0x7f1a5c0749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 -- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a5c1020d0 msgr2=0x7f1a5c075e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a5c1020d0 0x7f1a5c075e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 -- 172.21.15.186:0/2890789083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c0095f0 con 0x7f1a5c1032d0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a5a7fc700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 0x7f1a5c074470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000d980 tx=0x7f1a5000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50004220 con 0x7f1a5c1032d0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c1a18d0 con 0x7f1a5c1032d0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1a50007640 con 0x7f1a5c1032d0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50005520 con 0x7f1a5c1032d0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.762+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c1a1e20 con 0x7f1a5c1032d0 2024-03-26T04:45:38.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.763+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c109510 con 0x7f1a5c1032d0 2024-03-26T04:45:38.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.764+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) v1 ==== 91449+0+0 (secure 0 0 0) 0x7f1a50004750 con 0x7f1a5c1032d0 2024-03-26T04:45:38.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.764+0000 7f1a43fff700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1a4406da60 0x7f1a4406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:38.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.764+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f1a5008d530 con 0x7f1a5c1032d0 2024-03-26T04:45:38.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.764+0000 7f1a5affd700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1a4406da60 0x7f1a4406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:38.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.765+0000 7f1a5affd700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1a4406da60 0x7f1a4406ff20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00b580 tx=0x7f1a4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:38.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.769+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1a500569b0 con 0x7f1a5c1032d0 2024-03-26T04:45:38.981 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.981+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi012=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a5c10d1a0 con 0x7f1a4406da60 2024-03-26T04:45:38.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.989+0000 7f1a43fff700 1 -- 172.21.15.186:0/2890789083 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f1a5c10d1a0 con 0x7f1a4406da60 2024-03-26T04:45:38.989 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled alertmanager update... 2024-03-26T04:45:38.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1a4406da60 msgr2=0x7f1a4406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:38.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1a4406da60 0x7f1a4406ff20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00b580 tx=0x7f1a4c005fb0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.990 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 msgr2=0x7f1a5c074470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 0x7f1a5c074470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000d980 tx=0x7f1a5000dc90 comp rx=0 tx=0).stop 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 shutdown_connections 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f1a4406da60 0x7f1a4406ff20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a5c1020d0 0x7f1a5c075e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1a5c103c70 0x7f1a5c0749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 --2- 172.21.15.186:0/2890789083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a5c1032d0 0x7f1a5c074470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:38.991 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 >> 172.21.15.186:0/2890789083 conn(0x7f1a5c0fd690 msgr2=0x7f1a5c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 shutdown_connections 2024-03-26T04:45:38.992 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:38.991+0000 7f1a61692700 1 -- 172.21.15.186:0/2890789083 wait complete. 2024-03-26T04:45:39.368 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:39 smithi186 ceph-mon[33438]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 161 op/s 2024-03-26T04:45:39.368 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:39 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:39.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:39 smithi012 ceph-mon[41141]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 161 op/s 2024-03-26T04:45:39.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:39 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:39.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:39 smithi012 ceph-mon[29863]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 161 op/s 2024-03-26T04:45:39.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:39 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:39.804 DEBUG:teuthology.orchestra.run.smithi012:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T04:45:39.807 INFO:tasks.cephadm:Adding grafana.a on smithi186 2024-03-26T04:45:39.807 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi186=a' 2024-03-26T04:45:40.044 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:40.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:40 smithi012 ceph-mon[29863]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi012=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:40.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:40 smithi012 ceph-mon[29863]: Saving service alertmanager spec with placement smithi012=a;count:1 2024-03-26T04:45:40.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:40 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:40.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:40 smithi012 ceph-mon[41141]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi012=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:40.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:40 smithi012 ceph-mon[41141]: Saving service alertmanager spec with placement smithi012=a;count:1 2024-03-26T04:45:40.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:40 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:40.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:40 smithi186 ceph-mon[33438]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi012=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:40.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:40 smithi186 ceph-mon[33438]: Saving service alertmanager spec with placement smithi012=a;count:1 2024-03-26T04:45:40.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:40 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:41.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:41 smithi012 ceph-mon[29863]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 2.5 KiB/s wr, 108 op/s 2024-03-26T04:45:41.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:41 smithi012 ceph-mon[41141]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 2.5 KiB/s wr, 108 op/s 2024-03-26T04:45:41.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:41 smithi186 ceph-mon[33438]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 2.5 KiB/s wr, 108 op/s 2024-03-26T04:45:42.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:41 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:45:42.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.657+0000 7f0eddeda700 1 -- 172.21.15.186:0/3547082811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8107da0 msgr2=0x7f0ed810a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.657+0000 7f0eddeda700 1 --2- 172.21.15.186:0/3547082811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8107da0 0x7f0ed810a210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b0a0 tx=0x7f0ed000b3b0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 -- 172.21.15.186:0/3547082811 shutdown_connections 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 --2- 172.21.15.186:0/3547082811 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed8102b40 0x7f0ed8104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 --2- 172.21.15.186:0/3547082811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8107da0 0x7f0ed810a210 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 --2- 172.21.15.186:0/3547082811 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ed8105470 0x7f0ed8107860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 -- 172.21.15.186:0/3547082811 >> 172.21.15.186:0/3547082811 conn(0x7f0ed80fc570 msgr2=0x7f0ed80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 -- 172.21.15.186:0/3547082811 shutdown_connections 2024-03-26T04:45:42.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 -- 172.21.15.186:0/3547082811 wait complete. 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 Processor -- start 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 -- start start 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.658+0000 7f0eddeda700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed8102b40 0x7f0ed81024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 0x7f0ed8100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ed8107da0 0x7f0ed8101030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed81017b0 con 0x7f0ed8102b40 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8101920 con 0x7f0ed8105470 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed81a1870 con 0x7f0ed8107da0 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 0x7f0ed8100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 0x7f0ed8100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36130/0 (socket says 172.21.15.186:36130) 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 -- 172.21.15.186:0/1507675113 learned_addr learned my addr 172.21.15.186:0/1507675113 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:42.771 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed77fe700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed8102b40 0x7f0ed81024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed7fff700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ed8107da0 0x7f0ed8101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 -- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ed8107da0 msgr2=0x7f0ed8101030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ed8107da0 0x7f0ed8101030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 -- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed8102b40 msgr2=0x7f0ed81024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed8102b40 0x7f0ed81024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 -- 172.21.15.186:0/1507675113 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed0009d00 con 0x7f0ed8105470 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed6ffd700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 0x7f0ed8100af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b530 tx=0x7f0ec800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8011830 con 0x7f0ed8105470 2024-03-26T04:45:42.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0ec8011e10 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed81a1b50 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8010e30 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.659+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed81a2050 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.660+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed80fe0f0 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.661+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) v1 ==== 91449+0+0 (secure 0 0 0) 0x7f0ec8011990 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.661+0000 7f0ed4ff9700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0ec006da60 0x7f0ec006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.661+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f0ec808d250 con 0x7f0ed8105470 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.661+0000 7f0ed77fe700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0ec006da60 0x7f0ec006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.662+0000 7f0ed77fe700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0ec006da60 0x7f0ec006ff20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009a10 tx=0x7f0ecc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:42.773 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.666+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0ec80567e0 con 0x7f0ed8105470 2024-03-26T04:45:42.880 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.880+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 --> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ed80624c0 con 0x7f0ec006da60 2024-03-26T04:45:42.880 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:42 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:42.881 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:42 smithi186 ceph-mon[33438]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.2 KiB/s wr, 54 op/s 2024-03-26T04:45:42.889 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.889+0000 7f0ed4ff9700 1 -- 172.21.15.186:0/1507675113 <== mgr.14152 v2:172.21.15.12:6800/1454619140 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0ed80624c0 con 0x7f0ec006da60 2024-03-26T04:45:42.889 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled grafana update... 2024-03-26T04:45:42.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0ec006da60 msgr2=0x7f0ec006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:42.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0ec006da60 0x7f0ec006ff20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009a10 tx=0x7f0ecc009190 comp rx=0 tx=0).stop 2024-03-26T04:45:42.891 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 msgr2=0x7f0ed8100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 0x7f0ed8100af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b530 tx=0x7f0ec800b840 comp rx=0 tx=0).stop 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 shutdown_connections 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f0ec006da60 0x7f0ec006ff20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ed8102b40 0x7f0ed81024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0ed8107da0 0x7f0ed8101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 --2- 172.21.15.186:0/1507675113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed8105470 0x7f0ed8100af0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 >> 172.21.15.186:0/1507675113 conn(0x7f0ed80fc570 msgr2=0x7f0ed81037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 shutdown_connections 2024-03-26T04:45:42.892 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:42.891+0000 7f0eddeda700 1 -- 172.21.15.186:0/1507675113 wait complete. 2024-03-26T04:45:43.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:42 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:43.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:42 smithi012 ceph-mon[29863]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.2 KiB/s wr, 54 op/s 2024-03-26T04:45:43.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:42 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:43.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:42 smithi012 ceph-mon[41141]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 1.2 KiB/s wr, 54 op/s 2024-03-26T04:45:44.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:43 smithi012 ceph-mon[41141]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:44.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:43 smithi012 ceph-mon[41141]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-26T04:45:44.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:43 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:44.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:43 smithi012 ceph-mon[29863]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:44.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:43 smithi012 ceph-mon[29863]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-26T04:45:44.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:43 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:44.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:43 smithi186 ceph-mon[33438]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:44.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:43 smithi186 ceph-mon[33438]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-26T04:45:44.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:43 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:44.677 DEBUG:teuthology.orchestra.run.smithi186:grafana.a> sudo journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T04:45:44.679 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T04:45:44.679 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:45:44.941 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-26T04:45:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-26T04:45:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:45:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:44 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-26T04:45:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:44 smithi186 ceph-mon[33438]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:45:45.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-26T04:45:45.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-26T04:45:45.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[29863]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-26T04:45:45.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:44 smithi012 ceph-mon[41141]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:45:46.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-26T04:45:46.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-26T04:45:46.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T04:45:46.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-26T04:45:46.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[29863]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:45:46.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-26T04:45:46.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-26T04:45:46.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T04:45:46.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-26T04:45:46.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:45 smithi012 ceph-mon[41141]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:45:46.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-26T04:45:46.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-03-26T04:45:46.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T04:45:46.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:45 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-26T04:45:46.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:45 smithi186 ceph-mon[33438]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:45:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 -- 172.21.15.12:0/1488545545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8102250 msgr2=0x7f90c81026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 --2- 172.21.15.12:0/1488545545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8102250 0x7f90c81026b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0099c0 tx=0x7f90bc009cd0 comp rx=0 tx=0).stop 2024-03-26T04:45:46.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 -- 172.21.15.12:0/1488545545 shutdown_connections 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 --2- 172.21.15.12:0/1488545545 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 0x7f90c81094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 --2- 172.21.15.12:0/1488545545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90c8101050 0x7f90c8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 --2- 172.21.15.12:0/1488545545 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8102250 0x7f90c81026b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 -- 172.21.15.12:0/1488545545 >> 172.21.15.12:0/1488545545 conn(0x7f90c80fc610 msgr2=0x7f90c80fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 -- 172.21.15.12:0/1488545545 shutdown_connections 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.772+0000 7f90ccde0700 1 -- 172.21.15.12:0/1488545545 wait complete. 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 Processor -- start 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 -- start start 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 0x7f90c81a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90c8102250 0x7f90c81a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 0x7f90c81a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c81a56a0 con 0x7f90c8101050 2024-03-26T04:45:46.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c81a5810 con 0x7f90c8102250 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90ccde0700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c81a59d0 con 0x7f90c8102bf0 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 0x7f90c81a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 0x7f90c81a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38398/0 (socket says 172.21.15.12:38398) 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 -- 172.21.15.12:0/3479987991 learned_addr learned my addr 172.21.15.12:0/3479987991 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c6d9d700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 0x7f90c81a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c5d9b700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90c8102250 0x7f90c81a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 -- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 msgr2=0x7f90c81a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 0x7f90c81a4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 -- 172.21.15.12:0/3479987991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90c8102250 msgr2=0x7f90c81a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90c8102250 0x7f90c81a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:46.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 -- 172.21.15.12:0/3479987991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c0009c40 con 0x7f90c8101050 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c6d9d700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 0x7f90c81a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.773+0000 7f90c659c700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 0x7f90c81a02c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90b800e970 tx=0x7f90b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.774+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8009790 con 0x7f90c8101050 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.774+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f90b800eeb0 con 0x7f90c8101050 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.774+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc0095f0 con 0x7f90c8101050 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.774+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8003b60 con 0x7f90c8101050 2024-03-26T04:45:46.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.774+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c81a6020 con 0x7f90c8101050 2024-03-26T04:45:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.775+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c8050590 con 0x7f90c8101050 2024-03-26T04:45:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.775+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) v1 ==== 91449+0+0 (secure 0 0 0) 0x7f90b8003d00 con 0x7f90c8101050 2024-03-26T04:45:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.776+0000 7f90b77fe700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f90b006db10 0x7f90b006ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:46.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.776+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7236+0+0 (secure 0 0 0) 0x7f90b8014070 con 0x7f90c8101050 2024-03-26T04:45:46.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.776+0000 7f90c5d9b700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f90b006db10 0x7f90b006ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.776+0000 7f90c5d9b700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f90b006db10 0x7f90b006ffd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00b580 tx=0x7f90bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:46.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:46.780+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f90b8056e90 con 0x7f90c8101050 2024-03-26T04:45:47.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.069+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f90c810c040 con 0x7f90c8101050 2024-03-26T04:45:47.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:46 smithi012 ceph-mon[29863]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:45:47.070 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:46 smithi012 ceph-mon[29863]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-26T04:45:47.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:46 smithi012 ceph-mon[41141]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:45:47.070 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:46 smithi012 ceph-mon[41141]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-26T04:45:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.071+0000 7f90b77fe700 1 -- 172.21.15.12:0/3479987991 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f90b805a550 con 0x7f90c8101050 2024-03-26T04:45:47.072 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-03-26T04:45:47.072 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQD7UgJmjH48BBAAyjyN5bY0Uf04vSMZHMBajA== 2024-03-26T04:45:47.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f90b006db10 msgr2=0x7f90b006ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f90b006db10 0x7f90b006ffd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00b580 tx=0x7f90bc005fb0 comp rx=0 tx=0).stop 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 msgr2=0x7f90c81a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 0x7f90c81a02c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90b800e970 tx=0x7f90b800ed30 comp rx=0 tx=0).stop 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 shutdown_connections 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f90b006db10 0x7f90b006ffd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90c8101050 0x7f90c81a02c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f90c8102bf0 0x7f90c81a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 --2- 172.21.15.12:0/3479987991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90c8102250 0x7f90c81a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.074+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 >> 172.21.15.12:0/3479987991 conn(0x7f90c80fc610 msgr2=0x7f90c80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.075+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 shutdown_connections 2024-03-26T04:45:47.075 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:47.075+0000 7f90ccde0700 1 -- 172.21.15.12:0/3479987991 wait complete. 2024-03-26T04:45:47.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:46 smithi186 ceph-mon[33438]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:45:47.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:46 smithi186 ceph-mon[33438]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-26T04:45:47.743 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:45:47.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T04:45:47.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T04:45:47.788 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:45:48.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3479987991' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:48.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3479987991' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:45:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3479987991' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3479987991' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:45:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3479987991' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:48.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3479987991' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:45:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:48 smithi186 ceph-mon[33438]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-26T04:45:49.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:48 smithi012 ceph-mon[29863]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-26T04:45:49.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:48 smithi012 ceph-mon[41141]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-26T04:45:49.423 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:45:50.707 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:50 smithi186 systemd[1]: Starting Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:45:50.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:50 smithi186 ceph-mon[33438]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 2 objects/s recovering 2024-03-26T04:45:51.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:50 smithi012 ceph-mon[29863]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 2 objects/s recovering 2024-03-26T04:45:51.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:50 smithi012 ceph-mon[41141]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 2 objects/s recovering 2024-03-26T04:45:51.365 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 podman[62216]: 2024-03-26 04:45:51.135295442 +0000 UTC m=+0.450617664 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T04:45:51.366 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 podman[62216]: 2024-03-26 04:45:51.257939686 +0000 UTC m=+0.573261896 container create 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:45:51.932 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:45:51.932 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 podman[62216]: 2024-03-26 04:45:51.722647192 +0000 UTC m=+1.037969401 container init 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:45:51.932 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 podman[62216]: 2024-03-26 04:45:51.725798097 +0000 UTC m=+1.041120306 container start 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:45:51.932 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.748Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T04:45:51.932 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.748Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.748Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi186 (none))" 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.748Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.748Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.750Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.750Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.751Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.751Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.752Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.752Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.483µs 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.752Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.753Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.753Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=15.184µs wal_replay_duration=591.336µs wbl_replay_duration=161ns total_replay_duration=623.217µs 2024-03-26T04:45:51.933 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.754Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.754Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.754Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.773Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.184453ms db_storage=1.204µs remote_storage=1.437µs web_handler=460ns query_engine=593ns scrape=443.299µs scrape_sd=65.063µs notify=1.064µs notify_sd=1.316µs rules=17.424738ms tracing=6.559µs 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.773Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:45:51.773Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 bash[62216]: 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 2024-03-26T04:45:51.934 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:45:51 smithi186 systemd[1]: Started Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:45:52.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 -- 172.21.15.186:0/3601555105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb460107dd0 msgr2=0x7fb46010a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:52.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 --2- 172.21.15.186:0/3601555105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb460107dd0 0x7fb46010a240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb45800b0a0 tx=0x7fb45800b3b0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 -- 172.21.15.186:0/3601555105 shutdown_connections 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 --2- 172.21.15.186:0/3601555105 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb460102b70 0x7fb460104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 --2- 172.21.15.186:0/3601555105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb460107dd0 0x7fb46010a240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 --2- 172.21.15.186:0/3601555105 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb4601054a0 0x7fb460107890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 -- 172.21.15.186:0/3601555105 >> 172.21.15.186:0/3601555105 conn(0x7fb4600fc5a0 msgr2=0x7fb4600fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 -- 172.21.15.186:0/3601555105 shutdown_connections 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 -- 172.21.15.186:0/3601555105 wait complete. 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.043+0000 7fb468580700 1 Processor -- start 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 -- start start 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb460102b70 0x7fb4601024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 0x7fb460100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb460107dd0 0x7fb460101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:52.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4601017e0 con 0x7fb460102b70 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460101950 con 0x7fb4601054a0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4601a1680 con 0x7fb460107dd0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 0x7fb460100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 0x7fb460100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46618/0 (socket says 172.21.15.186:46618) 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 -- 172.21.15.186:0/965138980 learned_addr learned my addr 172.21.15.186:0/965138980 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb466b1d700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb460107dd0 0x7fb460101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb46631c700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb460102b70 0x7fb4601024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 -- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb460107dd0 msgr2=0x7fb460101060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb460107dd0 0x7fb460101060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 -- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb460102b70 msgr2=0x7fb4601024d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb460102b70 0x7fb4601024d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 -- 172.21.15.186:0/965138980 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb458009d00 con 0x7fb4601054a0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb465b1b700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 0x7fb460100b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb45000d950 tx=0x7fb45000dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:52.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450009b40 con 0x7fb4601054a0 2024-03-26T04:45:52.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb450009ca0 con 0x7fb4601054a0 2024-03-26T04:45:52.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45000f3c0 con 0x7fb4601054a0 2024-03-26T04:45:52.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.044+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4601a1960 con 0x7fb4601054a0 2024-03-26T04:45:52.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.045+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4601a1eb0 con 0x7fb4601054a0 2024-03-26T04:45:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.046+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4600fe120 con 0x7fb4601054a0 2024-03-26T04:45:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.046+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) v1 ==== 91449+0+0 (secure 0 0 0) 0x7fb450010460 con 0x7fb4601054a0 2024-03-26T04:45:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.047+0000 7fb4577fe700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb44c06db10 0x7fb44c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.047+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7236+0+0 (secure 0 0 0) 0x7fb45008d730 con 0x7fb4601054a0 2024-03-26T04:45:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.047+0000 7fb46631c700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb44c06db10 0x7fb44c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:52.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.047+0000 7fb46631c700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb44c06db10 0x7fb44c06ffd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb45c005950 tx=0x7fb45c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:52.051 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.051+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb450056bd0 con 0x7fb4601054a0 2024-03-26T04:45:52.324 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.324+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb46010d100 con 0x7fb4601054a0 2024-03-26T04:45:52.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.329+0000 7fb4577fe700 1 -- 172.21.15.186:0/965138980 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb4500200f0 con 0x7fb4601054a0 2024-03-26T04:45:52.329 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-03-26T04:45:52.329 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAAUwJmlPNwExAAzsjvUiOeki+ybH6RirtNDQ== 2024-03-26T04:45:52.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb44c06db10 msgr2=0x7fb44c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:52.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb44c06db10 0x7fb44c06ffd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb45c005950 tx=0x7fb45c0058e0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 msgr2=0x7fb460100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 0x7fb460100b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb45000d950 tx=0x7fb45000dd10 comp rx=0 tx=0).stop 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 shutdown_connections 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7fb44c06db10 0x7fb44c06ffd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb460102b70 0x7fb4601024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fb460107dd0 0x7fb460101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 --2- 172.21.15.186:0/965138980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4601054a0 0x7fb460100b20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 >> 172.21.15.186:0/965138980 conn(0x7fb4600fc5a0 msgr2=0x7fb4601037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 shutdown_connections 2024-03-26T04:45:52.332 INFO:teuthology.orchestra.run.smithi186.stderr:2024-03-26T04:45:52.331+0000 7fb468580700 1 -- 172.21.15.186:0/965138980 wait complete. 2024-03-26T04:45:52.935 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.186:0/965138980' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mon[33438]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 2 objects/s recovering 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: ignoring --setuser ceph since I am not root 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: ignoring --setgroup ceph since I am not root 2024-03-26T04:45:52.936 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:52 smithi186 ceph-mgr[35328]: -- 172.21.15.186:0/1103744938 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557cc2d4a1a0 con 0x557cc2d20800 2024-03-26T04:45:53.052 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-26T04:45:53.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T04:45:53.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T04:45:53.095 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T04:45:53.095 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T04:45:53.095 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T04:45:53.119 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ignoring --setuser ceph since I am not root 2024-03-26T04:45:53.119 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ignoring --setgroup ceph since I am not root 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.186:0/965138980' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[29863]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 2 objects/s recovering 2024-03-26T04:45:53.120 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.186:0/965138980' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:45:53.121 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:52 smithi012 ceph-mon[41141]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 82 B/s, 2 objects/s recovering 2024-03-26T04:45:53.354 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:53.713 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:53 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:53.457+0000 7fde696da200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:45:53.713 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:53 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:53.552+0000 7fde696da200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:45:53.733 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:53.492+0000 7f797f0c4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:45:53.733 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:53.589+0000 7f797f0c4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:45:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:53 smithi012 ceph-mon[29863]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:45:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:53 smithi012 ceph-mon[29863]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T04:45:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:53 smithi012 ceph-mon[41141]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:45:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:53 smithi012 ceph-mon[41141]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T04:45:54.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:53 smithi186 ceph-mon[33438]: from='mgr.14152 172.21.15.12:0/2240741051' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:45:54.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:53 smithi186 ceph-mon[33438]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T04:45:54.463 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.195+0000 7fde696da200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:45:54.463 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.289+0000 7fde696da200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:45:54.464 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.432+0000 7fde696da200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:45:54.518 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:54.257+0000 7f797f0c4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:45:54.518 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:54.351+0000 7f797f0c4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:45:54.518 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:54.497+0000 7f797f0c4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:45:54.771 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:54.599+0000 7f797f0c4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:45:54.771 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:54.686+0000 7f797f0c4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:45:54.882 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.530+0000 7fde696da200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:45:54.882 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.615+0000 7fde696da200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:45:54.882 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.882+0000 7fde696da200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:45:55.050 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:54 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:54.961+0000 7f797f0c4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:45:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 -- 172.21.15.12:0/881704639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf881080b0 msgr2=0x7faf88108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 --2- 172.21.15.12:0/881704639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf881080b0 0x7faf88108490 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faf780099c0 tx=0x7faf78009cd0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 -- 172.21.15.12:0/881704639 shutdown_connections 2024-03-26T04:45:55.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 --2- 172.21.15.12:0/881704639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf881080b0 0x7faf88108490 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 --2- 172.21.15.12:0/881704639 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf88102a90 0x7faf8810eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 --2- 172.21.15.12:0/881704639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf881020d0 0x7faf88102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 -- 172.21.15.12:0/881704639 >> 172.21.15.12:0/881704639 conn(0x7faf880791b0 msgr2=0x7faf880795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:55.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 -- 172.21.15.12:0/881704639 shutdown_connections 2024-03-26T04:45:55.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 -- 172.21.15.12:0/881704639 wait complete. 2024-03-26T04:45:55.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.064+0000 7faf8cc37700 1 Processor -- start 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 -- start start 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 0x7faf88074380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf88102a90 0x7faf880748c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf881080b0 0x7faf88074e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf881a5e90 con 0x7faf88102a90 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf881a6000 con 0x7faf881080b0 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8cc37700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf881a61c0 con 0x7faf881020d0 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 0x7faf88074380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf85d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf88102a90 0x7faf880748c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 0x7faf88074380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60448/0 (socket says 172.21.15.12:60448) 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 -- 172.21.15.12:0/4022418009 learned_addr learned my addr 172.21.15.12:0/4022418009 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:45:55.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf86d9d700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf881080b0 0x7faf88074e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 -- 172.21.15.12:0/4022418009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf881080b0 msgr2=0x7faf88074e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf881080b0 0x7faf88074e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 -- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf88102a90 msgr2=0x7faf880748c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf88102a90 0x7faf880748c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf8659c700 1 -- 172.21.15.12:0/4022418009 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80009c40 con 0x7faf881020d0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.065+0000 7faf85d9b700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf88102a90 0x7faf880748c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.066+0000 7faf8659c700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 0x7faf88074380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b580 tx=0x7faf7800b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.066+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7801d070 con 0x7faf881020d0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.066+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf780095f0 con 0x7faf881020d0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.066+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faf78004c70 con 0x7faf881020d0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.066+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78003680 con 0x7faf881020d0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.066+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf881a66c0 con 0x7faf881020d0 2024-03-26T04:45:55.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.067+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf88103760 con 0x7faf881020d0 2024-03-26T04:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.067+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 17) v1 ==== 91463+0+0 (secure 0 0 0) 0x7faf780047f0 con 0x7faf881020d0 2024-03-26T04:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.068+0000 7faf777fe700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 0x7faf7006ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:55.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.068+0000 7faf85d9b700 1 -- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 msgr2=0x7faf7006ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1454619140 2024-03-26T04:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.068+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7236+0+0 (secure 0 0 0) 0x7faf7808e410 con 0x7faf881020d0 2024-03-26T04:45:55.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.068+0000 7faf85d9b700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 0x7faf7006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:45:55.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.073+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faf78057950 con 0x7faf881020d0 2024-03-26T04:45:55.173 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:54.992+0000 7fde696da200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:45:55.174 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:55 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:55.079+0000 7fde696da200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:45:55.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.269+0000 7faf85d9b700 1 -- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 msgr2=0x7faf7006ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1454619140 2024-03-26T04:45:55.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.269+0000 7faf85d9b700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 0x7faf7006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T04:45:55.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.333+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7faf88111570 con 0x7faf881020d0 2024-03-26T04:45:55.334 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:55.089+0000 7f797f0c4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:45:55.334 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:55.179+0000 7f797f0c4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:45:55.334 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:55.276+0000 7f797f0c4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:45:55.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.339+0000 7faf777fe700 1 -- 172.21.15.12:0/4022418009 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+174861 (secure 0 0 0) 0x7faf7805b010 con 0x7faf881020d0 2024-03-26T04:45:55.342 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:45:55.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 msgr2=0x7faf7006ff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:45:55.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 0x7faf7006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 msgr2=0x7faf88074380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 0x7faf88074380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b580 tx=0x7faf7800b9c0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 shutdown_connections 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7faf7006dab0 0x7faf7006ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf881080b0 0x7faf88074e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7faf881020d0 0x7faf88074380 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 --2- 172.21.15.12:0/4022418009 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf88102a90 0x7faf880748c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 >> 172.21.15.12:0/4022418009 conn(0x7faf880791b0 msgr2=0x7faf88105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.341+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 shutdown_connections 2024-03-26T04:45:55.344 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:55.342+0000 7faf8cc37700 1 -- 172.21.15.12:0/4022418009 wait complete. 2024-03-26T04:45:55.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4022418009' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:45:55.464 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:55 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:55.173+0000 7fde696da200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:45:55.464 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:55 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:55.352+0000 7fde696da200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:45:55.620 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4022418009' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:45:55.620 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4022418009' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:45:55.620 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:55 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:55.458+0000 7f797f0c4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:45:55.924 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":1454619140},{"type":"v1","addr":"172.21.15.12:6801","nonce":1454619140}]},"active_addr":"172.21.15.12:6801/1454619140","active_change":"2024-03-26T04:40:44.587256+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.12:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":668110518}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2903486482}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1876359905}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":860229015}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":4274829844}]}]} 2024-03-26T04:45:55.927 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T04:45:55.927 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T04:45:55.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:45:56.220 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:56.618 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:56.478+0000 7f797f0c4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:45:56.618 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:56.617+0000 7f797f0c4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:45:56.634 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:56.328+0000 7fde696da200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:45:56.634 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:56.465+0000 7fde696da200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:45:56.920 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:56.823+0000 7fde696da200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:45:57.213 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:56.920+0000 7fde696da200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:45:57.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:57.127+0000 7fde696da200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:45:57.264 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:56.960+0000 7f797f0c4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:45:57.264 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:57.053+0000 7f797f0c4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:45:57.264 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:57.263+0000 7f797f0c4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:45:57.602 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:57.321+0000 7fde696da200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:45:57.621 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:57.458+0000 7f797f0c4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:45:57.874 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:57 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:57.759+0000 7f797f0c4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:45:57.886 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:57.622+0000 7fde696da200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 -- 172.21.15.12:0/4183727877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 msgr2=0x7f216c10ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 --2- 172.21.15.12:0/4183727877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10ef90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f216400b0a0 tx=0x7f216400b3b0 comp rx=0 tx=0).stop 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 -- 172.21.15.12:0/4183727877 shutdown_connections 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 --2- 172.21.15.12:0/4183727877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10ef90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 --2- 172.21.15.12:0/4183727877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 0x7f216c1074d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 --2- 172.21.15.12:0/4183727877 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 -- 172.21.15.12:0/4183727877 >> 172.21.15.12:0/4183727877 conn(0x7f216c077580 msgr2=0x7f216c0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 -- 172.21.15.12:0/4183727877 shutdown_connections 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.874+0000 7f217114b700 1 -- 172.21.15.12:0/4183727877 wait complete. 2024-03-26T04:45:57.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 Processor -- start 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 -- start start 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c10dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 0x7f216c1a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c1a31e0 con 0x7f216c101030 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c1a3350 con 0x7f216c1070f0 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f217114b700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c1a3510 con 0x7f216c100670 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c10dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c10dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:60466/0 (socket says 172.21.15.12:60466) 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f216ad9d700 1 -- 172.21.15.12:0/2979079929 learned_addr learned my addr 172.21.15.12:0/2979079929 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f216a59c700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.875+0000 7f216b59e700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 0x7f216c1a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216b59e700 1 -- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 msgr2=0x7f216c10dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:57.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216b59e700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c10dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216b59e700 1 -- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 msgr2=0x7f216c10e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216b59e700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216b59e700 1 -- 172.21.15.12:0/2979079929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2164009d00 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216ad9d700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216a59c700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f216b59e700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 0x7f216c1a2bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f216400b070 tx=0x7f2164007930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216400e040 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f216401d470 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c1a3790 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21640052c0 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.876+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c1ab050 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.877+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c103990 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.877+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) v1 ==== 91463+0+0 (secure 0 0 0) 0x7f2164004a50 con 0x7f216c1070f0 2024-03-26T04:45:57.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.878+0000 7f2153fff700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 0x7f215406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:45:57.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.878+0000 7f216ad9d700 1 -- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 msgr2=0x7f215406ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1454619140 2024-03-26T04:45:57.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.878+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7236+0+0 (secure 0 0 0) 0x7f216408ebf0 con 0x7f216c1070f0 2024-03-26T04:45:57.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.878+0000 7f216ad9d700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 0x7f215406ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T04:45:57.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:57.883+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2164057f10 con 0x7f216c1070f0 2024-03-26T04:45:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.079+0000 7f216ad9d700 1 -- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 msgr2=0x7f215406ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1454619140 2024-03-26T04:45:58.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.079+0000 7f216ad9d700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 0x7f215406ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T04:45:58.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.086+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f216c10f030 con 0x7f216c1070f0 2024-03-26T04:45:58.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.088+0000 7f2153fff700 1 -- 172.21.15.12:0/2979079929 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+22784 (secure 0 0 0) 0x7f216405b5d0 con 0x7f216c1070f0 2024-03-26T04:45:58.088 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:45:58.088 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":70,"fsid":"c3b00422-eb2a-11ee-95d0-87774f69a715","created":"2024-03-26T04:39:44.325097+0000","modified":"2024-03-26T04:45:45.893631+0000","last_up_change":"2024-03-26T04:45:06.281615+0000","last_in_change":"2024-03-26T04:44:48.805023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:43:00.923769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T04:45:19.155392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T04:45:19.401903+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:45:21.070239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:45:23.009269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:45:25.089191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff69b85-c494-4958-862f-62bc607e0807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6803","nonce":3811666428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6805","nonce":3811666428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6809","nonce":3811666428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6807","nonce":3811666428}]},"public_addr":"172.21.15.12:6803/3811666428","cluster_addr":"172.21.15.12:6805/3811666428","heartbeat_back_addr":"172.21.15.12:6809/3811666428","heartbeat_front_addr":"172.21.15.12:6807/3811666428","state":["exists","up"]},{"osd":1,"uuid":"0dc7629c-67f1-498c-a7c8-434c9f9695c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6811","nonce":4077554022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6813","nonce":4077554022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6817","nonce":4077554022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6815","nonce":4077554022}]},"public_addr":"172.21.15.12:6811/4077554022","cluster_addr":"172.21.15.12:6813/4077554022","heartbeat_back_addr":"172.21.15.12:6817/4077554022","heartbeat_front_addr":"172.21.15.12:6815/4077554022","state":["exists","up"]},{"osd":2,"uuid":"185f5e96-6db7-41a5-9384-384dae329880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6819","nonce":4166122647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6821","nonce":4166122647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6825","nonce":4166122647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6823","nonce":4166122647}]},"public_addr":"172.21.15.12:6819/4166122647","cluster_addr":"172.21.15.12:6821/4166122647","heartbeat_back_addr":"172.21.15.12:6825/4166122647","heartbeat_front_addr":"172.21.15.12:6823/4166122647","state":["exists","up"]},{"osd":3,"uuid":"ec8405e3-d836-443e-98d6-998c0ceb0a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6827","nonce":3848141010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6829","nonce":3848141010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6833","nonce":3848141010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6831","nonce":3848141010}]},"public_addr":"172.21.15.12:6827/3848141010","cluster_addr":"172.21.15.12:6829/3848141010","heartbeat_back_addr":"172.21.15.12:6833/3848141010","heartbeat_front_addr":"172.21.15.12:6831/3848141010","state":["exists","up"]},{"osd":4,"uuid":"745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6801","nonce":3595955502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6803","nonce":3595955502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6807","nonce":3595955502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6805","nonce":3595955502}]},"public_addr":"172.21.15.186:6801/3595955502","cluster_addr":"172.21.15.186:6803/3595955502","heartbeat_back_addr":"172.21.15.186:6807/3595955502","heartbeat_front_addr":"172.21.15.186:6805/3595955502","state":["exists","up"]},{"osd":5,"uuid":"1f918fca-3303-4d6c-a0d5-81d952478987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6809","nonce":223280961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6811","nonce":223280961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6815","nonce":223280961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6813","nonce":223280961}]},"public_addr":"172.21.15.186:6809/223280961","cluster_addr":"172.21.15.186:6811/223280961","heartbeat_back_addr":"172.21.15.186:6815/223280961","heartbeat_front_addr":"172.21.15.186:6813/223280961","state":["exists","up"]},{"osd":6,"uuid":"bdc95ff2-55b2-4401-b0d7-d8b58beb916d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6817","nonce":2797487459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6819","nonce":2797487459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6823","nonce":2797487459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6821","nonce":2797487459}]},"public_addr":"172.21.15.186:6817/2797487459","cluster_addr":"172.21.15.186:6819/2797487459","heartbeat_back_addr":"172.21.15.186:6823/2797487459","heartbeat_front_addr":"172.21.15.186:6821/2797487459","state":["exists","up"]},{"osd":7,"uuid":"04551815-3856-4382-85e7-bd7b39e62bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6825","nonce":1742409046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6827","nonce":1742409046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6831","nonce":1742409046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6829","nonce":1742409046}]},"public_addr":"172.21.15.186:6825/1742409046","cluster_addr":"172.21.15.186:6827/1742409046","heartbeat_back_addr":"172.21.15.186:6831/1742409046","heartbeat_front_addr":"172.21.15.186:6829/1742409046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:09.131249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:33.743929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:58.585033+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:43:23.412593+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:43:47.048739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:44:12.322403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:44:38.578143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:45:05.219851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6801/284140021":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/1561816911":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/239771954":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1875274271":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1841124439":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1796197499":"2024-03-27T04:40:44.587143+0000","172.21.15.12:6800/2152405375":"2024-03-27T04:40:17.651952+0000","172.21.15.12:6800/284140021":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/865472781":"2024-03-27T04:40:44.587143+0000","172.21.15.12:6801/2152405375":"2024-03-27T04:40:17.651952+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:45:58.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 msgr2=0x7f215406ff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 0x7f215406ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 msgr2=0x7f216c1a2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 0x7f216c1a2bd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f216400b070 tx=0x7f2164007930 comp rx=0 tx=0).stop 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 shutdown_connections 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:6800/1454619140,v1:172.21.15.12:6801/1454619140] conn(0x7f215406dab0 0x7f215406ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f216c1070f0 0x7f216c1a2bd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f216c100670 0x7f216c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 --2- 172.21.15.12:0/2979079929 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f216c101030 0x7f216c10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 >> 172.21.15.12:0/2979079929 conn(0x7f216c077580 msgr2=0x7f216c104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 shutdown_connections 2024-03-26T04:45:58.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:45:58.090+0000 7f217114b700 1 -- 172.21.15.12:0/2979079929 wait complete. 2024-03-26T04:45:58.140 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:58 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:58.026+0000 7f797f0c4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:45:58.140 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:57.886+0000 7fde696da200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:45:58.140 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:58.127+0000 7fde696da200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:45:58.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2979079929' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:45:58.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2979079929' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:45:58.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:58 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:58.277+0000 7f797f0c4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:45:58.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2979079929' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:45:58.774 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T04:45:58.774 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:45:59.028 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:45:59.213 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:58.947+0000 7fde696da200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:45:59.405 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:59.142+0000 7f797f0c4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:45:59.842 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:59.523+0000 7fde696da200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:45:59.842 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:59.623+0000 7fde696da200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[29863]: Standby manager daemon x restarted 2024-03-26T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[29863]: Standby manager daemon x started 2024-03-26T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:46:00.028 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[29863]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[41141]: Standby manager daemon x restarted 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[41141]: Standby manager daemon x started 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:46:00.029 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-mon[41141]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:46:00.030 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:59.733+0000 7f797f0c4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:46:00.030 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:45:59 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:45:59.835+0000 7f797f0c4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:46:00.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-mon[33438]: Standby manager daemon x restarted 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-mon[33438]: Standby manager daemon x started 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-mon[33438]: from='mgr.? 172.21.15.186:0/1499328153' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:45:59.842+0000 7fde696da200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: [26/Mar/2024:04:45:59] ENGINE Bus STARTING 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: CherryPy Checker: 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: The Application mounted at '' has an empty config. 2024-03-26T04:46:00.214 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: 2024-03-26T04:46:00.215 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: [26/Mar/2024:04:45:59] ENGINE Serving on http://:::9283 2024-03-26T04:46:00.215 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 04:45:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x[35324]: [26/Mar/2024:04:45:59] ENGINE Bus STARTED 2024-03-26T04:46:00.293 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:46:00.066+0000 7f797f0c4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:46:00.293 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:00] ENGINE Bus STARTING 2024-03-26T04:46:00.574 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: CherryPy Checker: 2024-03-26T04:46:00.575 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: The Application mounted at '' has an empty config. 2024-03-26T04:46:00.575 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T04:46:00.575 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:00] ENGINE Serving on http://:::9283 2024-03-26T04:46:00.575 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:00] ENGINE Bus STARTED 2024-03-26T04:46:00.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 -- 172.21.15.12:0/2081520936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83381070f0 msgr2=0x7f83381074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:00.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 --2- 172.21.15.12:0/2081520936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83381070f0 0x7f83381074d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f83340098f0 tx=0x7f8334009c00 comp rx=0 tx=0).stop 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 -- 172.21.15.12:0/2081520936 shutdown_connections 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 --2- 172.21.15.12:0/2081520936 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f83381070f0 0x7f83381074d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 --2- 172.21.15.12:0/2081520936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8338101030 0x7f833810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 --2- 172.21.15.12:0/2081520936 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338100670 0x7f8338100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 -- 172.21.15.12:0/2081520936 >> 172.21.15.12:0/2081520936 conn(0x7f8338077580 msgr2=0x7f83380fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 -- 172.21.15.12:0/2081520936 shutdown_connections 2024-03-26T04:46:00.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.598+0000 7f8340267700 1 -- 172.21.15.12:0/2081520936 wait complete. 2024-03-26T04:46:00.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 Processor -- start 2024-03-26T04:46:00.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 -- start start 2024-03-26T04:46:00.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 0x7f833819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:00.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338101030 0x7f833819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:00.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83381070f0 0x7f8338197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83381980f0 con 0x7f8338100670 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8338198230 con 0x7f83381070f0 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f8340267700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83381983f0 con 0x7f8338101030 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f833e003700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 0x7f833819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f833d802700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338101030 0x7f833819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f833e003700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 0x7f833819d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53608/0 (socket says 172.21.15.12:53608) 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.599+0000 7f833e003700 1 -- 172.21.15.12:0/2556211199 learned_addr learned my addr 172.21.15.12:0/2556211199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e804700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83381070f0 0x7f8338197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e003700 1 -- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338101030 msgr2=0x7f833819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e003700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338101030 0x7f833819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e003700 1 -- 172.21.15.12:0/2556211199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83381070f0 msgr2=0x7f8338197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e003700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83381070f0 0x7f8338197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.602 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e003700 1 -- 172.21.15.12:0/2556211199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83280095f0 con 0x7f8338100670 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e804700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83381070f0 0x7f8338197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833d802700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338101030 0x7f833819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f833e003700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 0x7f833819d8d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8334005950 tx=0x7f833400f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334020070 con 0x7f8338100670 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8334009520 con 0x7f8338100670 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f833400bb20 con 0x7f8338100670 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83340052a0 con 0x7f8338100670 2024-03-26T04:46:00.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.600+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83381988f0 con 0x7f8338100670 2024-03-26T04:46:00.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.602+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8338068d40 con 0x7f8338100670 2024-03-26T04:46:00.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.602+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) v1 ==== 91080+0+0 (secure 0 0 0) 0x7f8334024030 con 0x7f8338100670 2024-03-26T04:46:00.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.602+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f8334092530 con 0x7f8338100670 2024-03-26T04:46:00.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.607+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f833405bce0 con 0x7f8338100670 2024-03-26T04:46:00.893 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.893+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83381117c0 con 0x7f8338100670 2024-03-26T04:46:00.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.896+0000 7f832f7fe700 1 -- 172.21.15.12:0/2556211199 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+23222 (secure 0 0 0) 0x7f833405f3a0 con 0x7f8338100670 2024-03-26T04:46:00.896 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:46:00.897 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":71,"fsid":"c3b00422-eb2a-11ee-95d0-87774f69a715","created":"2024-03-26T04:39:44.325097+0000","modified":"2024-03-26T04:46:00.069078+0000","last_up_change":"2024-03-26T04:45:06.281615+0000","last_in_change":"2024-03-26T04:44:48.805023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:43:00.923769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T04:45:19.155392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T04:45:19.401903+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:45:21.070239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:45:23.009269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:45:25.089191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ff69b85-c494-4958-862f-62bc607e0807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6803","nonce":3811666428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6805","nonce":3811666428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6809","nonce":3811666428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":3811666428},{"type":"v1","addr":"172.21.15.12:6807","nonce":3811666428}]},"public_addr":"172.21.15.12:6803/3811666428","cluster_addr":"172.21.15.12:6805/3811666428","heartbeat_back_addr":"172.21.15.12:6809/3811666428","heartbeat_front_addr":"172.21.15.12:6807/3811666428","state":["exists","up"]},{"osd":1,"uuid":"0dc7629c-67f1-498c-a7c8-434c9f9695c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6811","nonce":4077554022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6813","nonce":4077554022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6817","nonce":4077554022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":4077554022},{"type":"v1","addr":"172.21.15.12:6815","nonce":4077554022}]},"public_addr":"172.21.15.12:6811/4077554022","cluster_addr":"172.21.15.12:6813/4077554022","heartbeat_back_addr":"172.21.15.12:6817/4077554022","heartbeat_front_addr":"172.21.15.12:6815/4077554022","state":["exists","up"]},{"osd":2,"uuid":"185f5e96-6db7-41a5-9384-384dae329880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6819","nonce":4166122647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6821","nonce":4166122647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6825","nonce":4166122647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":4166122647},{"type":"v1","addr":"172.21.15.12:6823","nonce":4166122647}]},"public_addr":"172.21.15.12:6819/4166122647","cluster_addr":"172.21.15.12:6821/4166122647","heartbeat_back_addr":"172.21.15.12:6825/4166122647","heartbeat_front_addr":"172.21.15.12:6823/4166122647","state":["exists","up"]},{"osd":3,"uuid":"ec8405e3-d836-443e-98d6-998c0ceb0a2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6827","nonce":3848141010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6829","nonce":3848141010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6833","nonce":3848141010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":3848141010},{"type":"v1","addr":"172.21.15.12:6831","nonce":3848141010}]},"public_addr":"172.21.15.12:6827/3848141010","cluster_addr":"172.21.15.12:6829/3848141010","heartbeat_back_addr":"172.21.15.12:6833/3848141010","heartbeat_front_addr":"172.21.15.12:6831/3848141010","state":["exists","up"]},{"osd":4,"uuid":"745dc2b5-91f4-4dcb-bd5d-95bdaaff7cd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6801","nonce":3595955502}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6803","nonce":3595955502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6807","nonce":3595955502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3595955502},{"type":"v1","addr":"172.21.15.186:6805","nonce":3595955502}]},"public_addr":"172.21.15.186:6801/3595955502","cluster_addr":"172.21.15.186:6803/3595955502","heartbeat_back_addr":"172.21.15.186:6807/3595955502","heartbeat_front_addr":"172.21.15.186:6805/3595955502","state":["exists","up"]},{"osd":5,"uuid":"1f918fca-3303-4d6c-a0d5-81d952478987","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6809","nonce":223280961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6811","nonce":223280961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6815","nonce":223280961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":223280961},{"type":"v1","addr":"172.21.15.186:6813","nonce":223280961}]},"public_addr":"172.21.15.186:6809/223280961","cluster_addr":"172.21.15.186:6811/223280961","heartbeat_back_addr":"172.21.15.186:6815/223280961","heartbeat_front_addr":"172.21.15.186:6813/223280961","state":["exists","up"]},{"osd":6,"uuid":"bdc95ff2-55b2-4401-b0d7-d8b58beb916d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6817","nonce":2797487459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6819","nonce":2797487459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6823","nonce":2797487459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2797487459},{"type":"v1","addr":"172.21.15.186:6821","nonce":2797487459}]},"public_addr":"172.21.15.186:6817/2797487459","cluster_addr":"172.21.15.186:6819/2797487459","heartbeat_back_addr":"172.21.15.186:6823/2797487459","heartbeat_front_addr":"172.21.15.186:6821/2797487459","state":["exists","up"]},{"osd":7,"uuid":"04551815-3856-4382-85e7-bd7b39e62bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6825","nonce":1742409046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6827","nonce":1742409046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6831","nonce":1742409046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1742409046},{"type":"v1","addr":"172.21.15.186:6829","nonce":1742409046}]},"public_addr":"172.21.15.186:6825/1742409046","cluster_addr":"172.21.15.186:6827/1742409046","heartbeat_back_addr":"172.21.15.186:6831/1742409046","heartbeat_front_addr":"172.21.15.186:6829/1742409046","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:09.131249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:33.743929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:42:58.585033+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:43:23.412593+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:43:47.048739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:44:12.322403+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:44:38.578143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:45:05.219851+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/1876359905":"2024-03-27T04:46:00.069022+0000","172.21.15.12:6801/2152405375":"2024-03-27T04:40:17.651952+0000","172.21.15.12:6800/284140021":"2024-03-27T04:40:44.587143+0000","172.21.15.12:6800/2152405375":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/1796197499":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/865472781":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/860229015":"2024-03-27T04:46:00.069022+0000","172.21.15.12:0/668110518":"2024-03-27T04:46:00.069022+0000","172.21.15.12:0/1561816911":"2024-03-27T04:40:44.587143+0000","172.21.15.12:0/239771954":"2024-03-27T04:40:17.651952+0000","172.21.15.12:6801/284140021":"2024-03-27T04:40:44.587143+0000","172.21.15.12:6800/1454619140":"2024-03-27T04:46:00.069022+0000","172.21.15.12:6801/1454619140":"2024-03-27T04:46:00.069022+0000","172.21.15.12:0/1875274271":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/2903486482":"2024-03-27T04:46:00.069022+0000","172.21.15.12:0/1841124439":"2024-03-27T04:40:17.651952+0000","172.21.15.12:0/4274829844":"2024-03-27T04:46:00.069022+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 msgr2=0x7f833819d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 0x7f833819d8d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8334005950 tx=0x7f833400f990 comp rx=0 tx=0).stop 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 shutdown_connections 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8338100670 0x7f833819d8d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83381070f0 0x7f8338197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 --2- 172.21.15.12:0/2556211199 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f8338101030 0x7f833819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.898+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 >> 172.21.15.12:0/2556211199 conn(0x7f8338077580 msgr2=0x7f8338104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.899+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 shutdown_connections 2024-03-26T04:46:00.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:00.899+0000 7f8340267700 1 -- 172.21.15.12:0/2556211199 wait complete. 2024-03-26T04:46:01.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: Active manager daemon y restarted 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: Activating manager daemon y 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: mgrmap e19: y(active, starting, since 0.00847465s), standbys: x 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:46:01.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: Manager daemon y is now available 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:46:01.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: Active manager daemon y restarted 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: Activating manager daemon y 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: mgrmap e19: y(active, starting, since 0.00847465s), standbys: x 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:46:01.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: Manager daemon y is now available 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2556211199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.160 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:46:01.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:46:01.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2556211199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T04:46:01.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: Active manager daemon y restarted 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: Activating manager daemon y 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: mgrmap e19: y(active, starting, since 0.00847465s), standbys: x 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:46:01.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: Manager daemon y is now available 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:46:01.216 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2556211199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:46:01.488 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T04:46:01.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T04:46:01.489 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T04:46:01.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T04:46:01.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T04:46:01.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T04:46:01.491 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T04:46:01.491 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T04:46:02.185 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:02 smithi186 ceph-mon[33438]: mgrmap e20: y(active, since 1.01483s), standbys: x 2024-03-26T04:46:02.185 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:02 smithi186 ceph-mon[33438]: [26/Mar/2024:04:46:01] ENGINE Bus STARTING 2024-03-26T04:46:02.185 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:02 smithi186 ceph-mon[33438]: [26/Mar/2024:04:46:01] ENGINE Serving on http://172.21.15.12:8765 2024-03-26T04:46:02.186 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:02 smithi186 ceph-mon[33438]: [26/Mar/2024:04:46:01] ENGINE Serving on https://172.21.15.12:7150 2024-03-26T04:46:02.186 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:02 smithi186 ceph-mon[33438]: [26/Mar/2024:04:46:01] ENGINE Bus STARTED 2024-03-26T04:46:02.186 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:02 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:02.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:46:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:46:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[29863]: mgrmap e20: y(active, since 1.01483s), standbys: x 2024-03-26T04:46:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[29863]: [26/Mar/2024:04:46:01] ENGINE Bus STARTING 2024-03-26T04:46:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[29863]: [26/Mar/2024:04:46:01] ENGINE Serving on http://172.21.15.12:8765 2024-03-26T04:46:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[29863]: [26/Mar/2024:04:46:01] ENGINE Serving on https://172.21.15.12:7150 2024-03-26T04:46:02.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[29863]: [26/Mar/2024:04:46:01] ENGINE Bus STARTED 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[41141]: mgrmap e20: y(active, since 1.01483s), standbys: x 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[41141]: [26/Mar/2024:04:46:01] ENGINE Bus STARTING 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[41141]: [26/Mar/2024:04:46:01] ENGINE Serving on http://172.21.15.12:8765 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[41141]: [26/Mar/2024:04:46:01] ENGINE Serving on https://172.21.15.12:7150 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[41141]: [26/Mar/2024:04:46:01] ENGINE Bus STARTED 2024-03-26T04:46:02.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:02 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:02.688 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:02.692 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:02.693 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:03.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:03 smithi012 ceph-mon[29863]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:03.127 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:03 smithi012 ceph-mon[41141]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:03.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:03 smithi186 ceph-mon[33438]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:03.683 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:03.684 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:03.696 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:03.698 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:04 smithi012 ceph-mon[29863]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T04:46:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:04 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:04.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:04 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:04.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:04 smithi012 ceph-mon[41141]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T04:46:04.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:04 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:04.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:04 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:04.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:04 smithi186 ceph-mon[33438]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T04:46:04.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:04 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:04.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:04 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.486 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:05.486 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.486 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.486 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.486 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi186 to 3760M 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[41141]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.487 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi186 to 3760M 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.488 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:05 smithi012 ceph-mon[29863]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi186 to 3760M 2024-03-26T04:46:05.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:05.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:05 smithi186 ceph-mon[33438]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-26T04:46:06.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 -- 172.21.15.12:0/598398237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 msgr2=0x7f1b481073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:06.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 --2- 172.21.15.12:0/598398237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 0x7f1b481073a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009990 tx=0x7f1b38009ca0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 -- 172.21.15.12:0/598398237 shutdown_connections 2024-03-26T04:46:06.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 --2- 172.21.15.12:0/598398237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 0x7f1b4810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 --2- 172.21.15.12:0/598398237 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 0x7f1b481073a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 --2- 172.21.15.12:0/598398237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b48100f80 0x7f1b481013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 -- 172.21.15.12:0/598398237 >> 172.21.15.12:0/598398237 conn(0x7f1b48077eb0 msgr2=0x7f1b480782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 -- 172.21.15.12:0/598398237 shutdown_connections 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.454+0000 7f1b4e566700 1 -- 172.21.15.12:0/598398237 wait complete. 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 Processor -- start 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 -- start start 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b48100f80 0x7f1b48198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 0x7f1b48198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 0x7f1b481a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48199740 con 0x7f1b48101920 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b481a3f50 con 0x7f1b48100f80 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4e566700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b481a4090 con 0x7f1b48106fc0 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b477fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 0x7f1b48198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:06.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b477fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 0x7f1b48198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53640/0 (socket says 172.21.15.12:53640) 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b477fe700 1 -- 172.21.15.12:0/4109681406 learned_addr learned my addr 172.21.15.12:0/4109681406 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b4cb03700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 0x7f1b481a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.455+0000 7f1b47fff700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b48100f80 0x7f1b48198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b477fe700 1 -- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 msgr2=0x7f1b481a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b477fe700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 0x7f1b481a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b477fe700 1 -- 172.21.15.12:0/4109681406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b48100f80 msgr2=0x7f1b48198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:06.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b477fe700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b48100f80 0x7f1b48198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b477fe700 1 -- 172.21.15.12:0/4109681406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40009c40 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b4cb03700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1b48106fc0 0x7f1b481a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b47fff700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b48100f80 0x7f1b48198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b477fe700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 0x7f1b48198ed0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00c9b0 tx=0x7f1b3c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c007b00 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1b3c007c60 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b380095f0 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c018420 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.456+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b481a4690 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.457+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1b481107a0 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.457+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f1b3c00f450 con 0x7f1b48101920 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b457fa700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1b3006dac0 0x7f1b3006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:06.489 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f1b3c08dde0 con 0x7f1b48101920 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b47fff700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1b3006dac0 0x7f1b3006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b457fa700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] conn(0x7f1b300779d0 0x7f1b30079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 --> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b3007a540 con 0x7f1b300779d0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b3c08e230 con 0x7f1b48101920 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b4cb03700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] conn(0x7f1b300779d0 0x7f1b30079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.458+0000 7f1b47fff700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1b3006dac0 0x7f1b3006ff80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b3800b2e0 tx=0x7f1b38005ab0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.459+0000 7f1b4cb03700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] conn(0x7f1b300779d0 0x7f1b30079e70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.459+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== osd.2 v2:172.21.15.12:6818/4166122647 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1b3007a540 con 0x7f1b300779d0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.475+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 --> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1b48062710 con 0x7f1b300779d0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b457fa700 1 -- 172.21.15.12:0/4109681406 <== osd.2 v2:172.21.15.12:6818/4166122647 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1b48062710 con 0x7f1b300779d0 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] conn(0x7f1b300779d0 msgr2=0x7f1b30079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6818/4166122647,v1:172.21.15.12:6819/4166122647] conn(0x7f1b300779d0 0x7f1b30079e70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1b3006dac0 msgr2=0x7f1b3006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:06.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1b3006dac0 0x7f1b3006ff80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1b3800b2e0 tx=0x7f1b38005ab0 comp rx=0 tx=0).stop 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 msgr2=0x7f1b48198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 --2- 172.21.15.12:0/4109681406 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b48101920 0x7f1b48198ed0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00c9b0 tx=0x7f1b3c00cd70 comp rx=0 tx=0).stop 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4cb03700 1 -- 172.21.15.12:0/4109681406 reap_dead start 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 shutdown_connections 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 >> 172.21.15.12:0/4109681406 conn(0x7f1b48077eb0 msgr2=0x7f1b480ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 shutdown_connections 2024-03-26T04:46:06.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:06.476+0000 7f1b4e566700 1 -- 172.21.15.12:0/4109681406 wait complete. 2024-03-26T04:46:06.779 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:46:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:07.334 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:07 smithi012 ceph-mon[29863]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:07.334 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:07 smithi012 ceph-mon[41141]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:07 smithi186 ceph-mon[33438]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:08.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 -- 172.21.15.12:0/830717424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 msgr2=0x7f16bc1013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:08.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 --2- 172.21.15.12:0/830717424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc1013e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009990 tx=0x7f16ac009ca0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 -- 172.21.15.12:0/830717424 shutdown_connections 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 --2- 172.21.15.12:0/830717424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc101920 0x7f16bc10de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 --2- 172.21.15.12:0/830717424 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f16bc106fc0 0x7f16bc1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 --2- 172.21.15.12:0/830717424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc1013e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 -- 172.21.15.12:0/830717424 >> 172.21.15.12:0/830717424 conn(0x7f16bc077eb0 msgr2=0x7f16bc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 -- 172.21.15.12:0/830717424 shutdown_connections 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.491+0000 7f16c34a5700 1 -- 172.21.15.12:0/830717424 wait complete. 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 Processor -- start 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 -- start start 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc19d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f16bc101920 0x7f16bc19dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:08.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc106fc0 0x7f16bc197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc197fa0 con 0x7f16bc100f80 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc1980e0 con 0x7f16bc106fc0 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c34a5700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc1982a0 con 0x7f16bc101920 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c1241700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc19d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c0a40700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f16bc101920 0x7f16bc19dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c1241700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc19d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53656/0 (socket says 172.21.15.12:53656) 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c1241700 1 -- 172.21.15.12:0/78429151 learned_addr learned my addr 172.21.15.12:0/78429151 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.492+0000 7f16c1a42700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc106fc0 0x7f16bc197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c1241700 1 -- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f16bc101920 msgr2=0x7f16bc19dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:08.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c1241700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f16bc101920 0x7f16bc19dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c1241700 1 -- 172.21.15.12:0/78429151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc106fc0 msgr2=0x7f16bc197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c1241700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16bc106fc0 0x7f16bc197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c1241700 1 -- 172.21.15.12:0/78429151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0095f0 con 0x7f16bc100f80 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c1241700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc19d730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e9a0 tx=0x7f16b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8009790 con 0x7f16bc100f80 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f16b800ce90 con 0x7f16bc100f80 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc198580 con 0x7f16bc100f80 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8018420 con 0x7f16bc100f80 2024-03-26T04:46:08.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.493+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc102d30 con 0x7f16bc100f80 2024-03-26T04:46:08.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.494+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16bc110780 con 0x7f16bc100f80 2024-03-26T04:46:08.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f16b800ca10 con 0x7f16bc100f80 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16b27fc700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f16a806dac0 0x7f16a806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f16b8014070 con 0x7f16bc100f80 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16c0a40700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f16a806dac0 0x7f16a806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16b27fc700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] conn(0x7f16a8077a50 0x7f16a8079ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 --> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16a807a5c0 con 0x7f16a8077a50 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16b80922c0 con 0x7f16bc100f80 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.495+0000 7f16c1a42700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] conn(0x7f16a8077a50 0x7f16a8079ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.496+0000 7f16c0a40700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f16a806dac0 0x7f16a806ff80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b2e0 tx=0x7f16ac01c810 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.496+0000 7f16c1a42700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] conn(0x7f16a8077a50 0x7f16a8079ef0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:08.498 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.496+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== osd.0 v2:172.21.15.12:6802/3811666428 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f16a807a5c0 con 0x7f16a8077a50 2024-03-26T04:46:08.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 --> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f16bc105fd0 con 0x7f16a8077a50 2024-03-26T04:46:08.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16b27fc700 1 -- 172.21.15.12:0/78429151 <== osd.0 v2:172.21.15.12:6802/3811666428 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f16bc105fd0 con 0x7f16a8077a50 2024-03-26T04:46:08.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] conn(0x7f16a8077a50 msgr2=0x7f16a8079ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:08.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6802/3811666428,v1:172.21.15.12:6803/3811666428] conn(0x7f16a8077a50 0x7f16a8079ef0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f16a806dac0 msgr2=0x7f16a806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f16a806dac0 0x7f16a806ff80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00b2e0 tx=0x7f16ac01c810 comp rx=0 tx=0).stop 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 msgr2=0x7f16bc19d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 --2- 172.21.15.12:0/78429151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f16bc100f80 0x7f16bc19d730 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16b800e9a0 tx=0x7f16b800ecb0 comp rx=0 tx=0).stop 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c1a42700 1 -- 172.21.15.12:0/78429151 reap_dead start 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 shutdown_connections 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.513+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 >> 172.21.15.12:0/78429151 conn(0x7f16bc077eb0 msgr2=0x7f16bc069fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.514+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 shutdown_connections 2024-03-26T04:46:08.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:08.514+0000 7f16c34a5700 1 -- 172.21.15.12:0/78429151 wait complete. 2024-03-26T04:46:09.582 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:09 smithi012 ceph-mon[29863]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:46:09.582 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:09 smithi012 ceph-mon[41141]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:46:09.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:09 smithi186 ceph-mon[33438]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:46:10.428 INFO:teuthology.orchestra.run.smithi012.stdout:34359738418 2024-03-26T04:46:10.428 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T04:46:10.812 INFO:teuthology.orchestra.run.smithi012.stdout:77309411367 2024-03-26T04:46:10.812 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T04:46:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:11 smithi012 ceph-mon[29863]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:46:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:11 smithi012 ceph-mon[41141]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:46:11.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:11 smithi186 ceph-mon[33438]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:46:12.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:46:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:46:12.570 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:12 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:12.570 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:12 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:12.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:12 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:13.314 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:13 smithi012 ceph-mon[29863]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:46:13.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:13 smithi012 ceph-mon[41141]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:46:13.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:13 smithi186 ceph-mon[33438]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:46:14.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 -- 172.21.15.12:0/1241052355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c0107360 msgr2=0x7f57c010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 --2- 172.21.15.12:0/1241052355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c0107360 0x7f57c010ef30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009f90 tx=0x7f57b800b330 comp rx=0 tx=0).stop 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 -- 172.21.15.12:0/1241052355 shutdown_connections 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 --2- 172.21.15.12:0/1241052355 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c0107360 0x7f57c010ef30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 --2- 172.21.15.12:0/1241052355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c006a820 0x7f57c006ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 --2- 172.21.15.12:0/1241052355 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c006b1d0 0x7f57c0106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 -- 172.21.15.12:0/1241052355 >> 172.21.15.12:0/1241052355 conn(0x7f57c0076fd0 msgr2=0x7f57c00773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 -- 172.21.15.12:0/1241052355 shutdown_connections 2024-03-26T04:46:14.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 -- 172.21.15.12:0/1241052355 wait complete. 2024-03-26T04:46:14.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 Processor -- start 2024-03-26T04:46:14.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 -- start start 2024-03-26T04:46:14.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c006a820 0x7f57c019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.200+0000 7f57c859e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c006b1d0 0x7f57c019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c859e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c0107360 0x7f57c0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c859e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01980d0 con 0x7f57c0107360 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c859e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0198210 con 0x7f57c006b1d0 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c859e700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c01983d0 con 0x7f57c006a820 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c006a820 0x7f57c019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c6b3b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c0107360 0x7f57c0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c006a820 0x7f57c019d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48624/0 (socket says 172.21.15.12:48624) 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 -- 172.21.15.12:0/2205568419 learned_addr learned my addr 172.21.15.12:0/2205568419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c6b3b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c0107360 0x7f57c0197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43484/0 (socket says 172.21.15.12:43484) 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c5b39700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c006b1d0 0x7f57c019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 -- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c006b1d0 msgr2=0x7f57c019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c006b1d0 0x7f57c019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 -- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c0107360 msgr2=0x7f57c0197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c0107360 0x7f57c0197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 -- 172.21.15.12:0/2205568419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b00095f0 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c5b39700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c006b1d0 0x7f57c019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c6b3b700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57c0107360 0x7f57c0197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c633a700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c006a820 0x7f57c019d860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c920 tx=0x7f57bc00cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc007b00 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b8009c40 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f57bc00ce70 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc005350 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.201+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c01a4730 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.202+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57c0111850 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f57bc0054d0 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57b77fe700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f57ac06dac0 0x7f57ac06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f57bc08d510 con 0x7f57c006a820 2024-03-26T04:46:14.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57c5b39700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f57ac06dac0 0x7f57ac06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57b77fe700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] conn(0x7f57ac0779d0 0x7f57ac079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 --> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57ac07a540 con 0x7f57ac0779d0 2024-03-26T04:46:14.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.203+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57bc08fbd0 con 0x7f57c006a820 2024-03-26T04:46:14.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.204+0000 7f57c5b39700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f57ac06dac0 0x7f57ac06ff80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57c0106fa0 tx=0x7f57b0009310 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:14.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.204+0000 7f57c6b3b700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] conn(0x7f57ac0779d0 0x7f57ac079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.204+0000 7f57c6b3b700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] conn(0x7f57ac0779d0 0x7f57ac079e70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:14.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.205+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== osd.5 v2:172.21.15.186:6808/223280961 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f57ac07a540 con 0x7f57ac0779d0 2024-03-26T04:46:14.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.221+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 --> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57c00ff780 con 0x7f57ac0779d0 2024-03-26T04:46:14.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57b77fe700 1 -- 172.21.15.12:0/2205568419 <== osd.5 v2:172.21.15.186:6808/223280961 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f57c00ff780 con 0x7f57ac0779d0 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] conn(0x7f57ac0779d0 msgr2=0x7f57ac079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c859e700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.186:6808/223280961,v1:172.21.15.186:6809/223280961] conn(0x7f57ac0779d0 0x7f57ac079e70 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f57ac06dac0 msgr2=0x7f57ac06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c859e700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f57ac06dac0 0x7f57ac06ff80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f57c0106fa0 tx=0x7f57b0009310 comp rx=0 tx=0).stop 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c006a820 msgr2=0x7f57c019d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c859e700 1 --2- 172.21.15.12:0/2205568419 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f57c006a820 0x7f57c019d860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c920 tx=0x7f57bc00cc30 comp rx=0 tx=0).stop 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.222+0000 7f57c6b3b700 1 -- 172.21.15.12:0/2205568419 reap_dead start 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.223+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 shutdown_connections 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.223+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 >> 172.21.15.12:0/2205568419 conn(0x7f57c0076fd0 msgr2=0x7f57c00fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.223+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 shutdown_connections 2024-03-26T04:46:14.224 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.223+0000 7f57c859e700 1 -- 172.21.15.12:0/2205568419 wait complete. 2024-03-26T04:46:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 -- 172.21.15.12:0/3478430224 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f001006b120 msgr2=0x7f001010ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 --2- 172.21.15.12:0/3478430224 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f001006b120 0x7f001010ef40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0008009920 tx=0x7f0008009c30 comp rx=0 tx=0).stop 2024-03-26T04:46:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 -- 172.21.15.12:0/3478430224 shutdown_connections 2024-03-26T04:46:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 --2- 172.21.15.12:0/3478430224 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f001006b120 0x7f001010ef40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 --2- 172.21.15.12:0/3478430224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0010108100 0x7f00101084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 --2- 172.21.15.12:0/3478430224 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006a780 0x7f001006abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 -- 172.21.15.12:0/3478430224 >> 172.21.15.12:0/3478430224 conn(0x7f0010076fa0 msgr2=0x7f00100773b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 -- 172.21.15.12:0/3478430224 shutdown_connections 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.673+0000 7f0017388700 1 -- 172.21.15.12:0/3478430224 wait complete. 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 Processor -- start 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 -- start start 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f001006a780 0x7f001019d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006b120 0x7f001019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0010108100 0x7f0010197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010197fd0 con 0x7f001006b120 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010198110 con 0x7f001006a780 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0017388700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00101982d0 con 0x7f0010108100 2024-03-26T04:46:14.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0015925700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0010108100 0x7f0010197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006b120 0x7f001019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006b120 0x7f001019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43464/0 (socket says 172.21.15.12:43464) 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 -- 172.21.15.12:0/3874880345 learned_addr learned my addr 172.21.15.12:0/3874880345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0015925700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0010108100 0x7f0010197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48604/0 (socket says 172.21.15.12:48604) 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0015124700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f001006a780 0x7f001019d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 -- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0010108100 msgr2=0x7f0010197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0010108100 0x7f0010197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 -- 172.21.15.12:0/3874880345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f001006a780 msgr2=0x7f001019d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f001006a780 0x7f001019d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0014923700 1 -- 172.21.15.12:0/3874880345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00040095f0 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0015124700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f001006a780 0x7f001019d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.674+0000 7f0015925700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0010108100 0x7f0010197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.675+0000 7f0014923700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006b120 0x7f001019dcf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f000400ec10 tx=0x7f000400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.675+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000400cd40 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.675+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f00040044a0 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.675+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0008009520 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.675+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004010430 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.675+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010100fb0 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.676+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f00101118b0 con 0x7f001006b120 2024-03-26T04:46:14.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.676+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f0004003f50 con 0x7f001006b120 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f000e7fc700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efff806dac0 0x7efff806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f0004014070 con 0x7f001006b120 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f0015124700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efff806dac0 0x7efff806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f000e7fc700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] conn(0x7efff80779d0 0x7efff8079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 --> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efff807a540 con 0x7efff80779d0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00040902c0 con 0x7f001006b120 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f0015925700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] conn(0x7efff80779d0 0x7efff8079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f0015124700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efff806dac0 0x7efff806ff80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0010198ab0 tx=0x7f000000b040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.677+0000 7f0015925700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] conn(0x7efff80779d0 0x7efff8079e70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.678+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== osd.3 v2:172.21.15.12:6826/3848141010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efff807a540 con 0x7efff80779d0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.694+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 --> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0010107620 con 0x7efff80779d0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f000e7fc700 1 -- 172.21.15.12:0/3874880345 <== osd.3 v2:172.21.15.12:6826/3848141010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0010107620 con 0x7efff80779d0 2024-03-26T04:46:14.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] conn(0x7efff80779d0 msgr2=0x7efff8079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6826/3848141010,v1:172.21.15.12:6827/3848141010] conn(0x7efff80779d0 0x7efff8079e70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efff806dac0 msgr2=0x7efff806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efff806dac0 0x7efff806ff80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0010198ab0 tx=0x7f000000b040 comp rx=0 tx=0).stop 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006b120 msgr2=0x7f001019dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 --2- 172.21.15.12:0/3874880345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f001006b120 0x7f001019dcf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f000400ec10 tx=0x7f000400ef20 comp rx=0 tx=0).stop 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0015925700 1 -- 172.21.15.12:0/3874880345 reap_dead start 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 shutdown_connections 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 >> 172.21.15.12:0/3874880345 conn(0x7f0010076fa0 msgr2=0x7f0010105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 shutdown_connections 2024-03-26T04:46:14.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:12.695+0000 7f0017388700 1 -- 172.21.15.12:0/3874880345 wait complete. 2024-03-26T04:46:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 -- 172.21.15.12:0/2031730383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281011d0 msgr2=0x7f99281015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2031730383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281011d0 0x7f99281015b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f99180098f0 tx=0x7f9918009c00 comp rx=0 tx=0).stop 2024-03-26T04:46:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 -- 172.21.15.12:0/2031730383 shutdown_connections 2024-03-26T04:46:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2031730383 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99281011d0 0x7f99281015b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:15.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2031730383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9928106290 0x7f992810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:15.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2031730383 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9928101b80 0x7f9928105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:15.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 -- 172.21.15.12:0/2031730383 >> 172.21.15.12:0/2031730383 conn(0x7f9928077ef0 msgr2=0x7f9928078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:15.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 -- 172.21.15.12:0/2031730383 shutdown_connections 2024-03-26T04:46:15.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.173+0000 7f992e4e2700 1 -- 172.21.15.12:0/2031730383 wait complete. 2024-03-26T04:46:15.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 Processor -- start 2024-03-26T04:46:15.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 -- start start 2024-03-26T04:46:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99281011d0 0x7f992819d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9928101b80 0x7f992819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9928106290 0x7f9928197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:15.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9928197ff0 con 0x7f9928106290 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9928198130 con 0x7f9928101b80 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992e4e2700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99281982f0 con 0x7f99281011d0 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f9927fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99281011d0 0x7f992819d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f992ca7f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9928106290 0x7f9928197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f9927fff700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99281011d0 0x7f992819d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48658/0 (socket says 172.21.15.12:48658) 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f9927fff700 1 -- 172.21.15.12:0/2847069268 learned_addr learned my addr 172.21.15.12:0/2847069268 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.174+0000 7f99277fe700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9928101b80 0x7f992819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992ca7f700 1 -- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99281011d0 msgr2=0x7f992819d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992ca7f700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99281011d0 0x7f992819d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:15.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992ca7f700 1 -- 172.21.15.12:0/2847069268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9928101b80 msgr2=0x7f992819dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992ca7f700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9928101b80 0x7f992819dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992ca7f700 1 -- 172.21.15.12:0/2847069268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c0095f0 con 0x7f9928106290 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f9927fff700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f99281011d0 0x7f992819d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f99277fe700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9928101b80 0x7f992819dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992ca7f700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9928106290 0x7f9928197990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f992000ab40 tx=0x7f992000ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992000f070 con 0x7f9928106290 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9920007a70 con 0x7f9928106290 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9918009520 con 0x7f9928106290 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920005280 con 0x7f9928106290 2024-03-26T04:46:15.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.175+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99281a46a0 con 0x7f9928106290 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.176+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f99281118d0 con 0x7f9928106290 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.176+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f992000c640 con 0x7f9928106290 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99257fa700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f991006dac0 0x7f991006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f992008cdb0 con 0x7f9928106290 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f9927fff700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f991006dac0 0x7f991006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99257fa700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] conn(0x7f99100779d0 0x7f9910079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 --> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f991007a540 con 0x7f99100779d0 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f992008f470 con 0x7f9928106290 2024-03-26T04:46:15.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99277fe700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] conn(0x7f99100779d0 0x7f9910079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f9927fff700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f991006dac0 0x7f991006ff80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f99180094f0 tx=0x7f9918019040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.177+0000 7f99277fe700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] conn(0x7f99100779d0 0x7f9910079e70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:15.182 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.178+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== osd.1 v2:172.21.15.12:6810/4077554022 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f991007a540 con 0x7f99100779d0 2024-03-26T04:46:15.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.200+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 --> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f99280ff740 con 0x7f99100779d0 2024-03-26T04:46:15.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f99257fa700 1 -- 172.21.15.12:0/2847069268 <== osd.1 v2:172.21.15.12:6810/4077554022 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f99280ff740 con 0x7f99100779d0 2024-03-26T04:46:15.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] conn(0x7f99100779d0 msgr2=0x7f9910079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:15.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6810/4077554022,v1:172.21.15.12:6811/4077554022] conn(0x7f99100779d0 0x7f9910079e70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f991006dac0 msgr2=0x7f991006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f991006dac0 0x7f991006ff80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f99180094f0 tx=0x7f9918019040 comp rx=0 tx=0).stop 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9928106290 msgr2=0x7f9928197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.201+0000 7f992e4e2700 1 --2- 172.21.15.12:0/2847069268 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9928106290 0x7f9928197990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f992000ab40 tx=0x7f992000ae50 comp rx=0 tx=0).stop 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.202+0000 7f992ca7f700 1 -- 172.21.15.12:0/2847069268 reap_dead start 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.202+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 shutdown_connections 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.202+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 >> 172.21.15.12:0/2847069268 conn(0x7f9928077ef0 msgr2=0x7f99280fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.202+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 shutdown_connections 2024-03-26T04:46:15.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.202+0000 7f992e4e2700 1 -- 172.21.15.12:0/2847069268 wait complete. 2024-03-26T04:46:15.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:15 smithi012 ceph-mon[29863]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:15.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:15 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:15.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:15 smithi012 ceph-mon[41141]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:15.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:15 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:15.651 INFO:teuthology.orchestra.run.smithi012.stdout:163208757274 2024-03-26T04:46:15.651 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T04:46:15.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:15 smithi186 ceph-mon[33438]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:15.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:15 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 -- 172.21.15.12:0/2295228411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f41073c0 msgr2=0x7fe8f410ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/2295228411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f41073c0 0x7fe8f410ef90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00b0a0 tx=0x7fe8ec00b3b0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 -- 172.21.15.12:0/2295228411 shutdown_connections 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/2295228411 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f41073c0 0x7fe8f410ef90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/2295228411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8f41022e0 0x7fe8f41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/2295228411 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f4102c90 0x7fe8f4106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 -- 172.21.15.12:0/2295228411 >> 172.21.15.12:0/2295228411 conn(0x7fe8f40fdb50 msgr2=0x7fe8f40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 -- 172.21.15.12:0/2295228411 shutdown_connections 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.851+0000 7fe8f8eac700 1 -- 172.21.15.12:0/2295228411 wait complete. 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.852+0000 7fe8f8eac700 1 Processor -- start 2024-03-26T04:46:16.289 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.852+0000 7fe8f8eac700 1 -- start start 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.852+0000 7fe8f8eac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8f41022e0 0x7fe8f419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.852+0000 7fe8f8eac700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f4102c90 0x7fe8f419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f8eac700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f41073c0 0x7fe8f4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f8eac700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f41980f0 con 0x7fe8f4102c90 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f8eac700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4198230 con 0x7fe8f41022e0 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f8eac700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f41983f0 con 0x7fe8f41073c0 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f41073c0 0x7fe8f4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f41073c0 0x7fe8f4197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48676/0 (socket says 172.21.15.12:48676) 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f1d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f4102c90 0x7fe8f419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 -- 172.21.15.12:0/1584914186 learned_addr learned my addr 172.21.15.12:0/1584914186 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f259c700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8f41022e0 0x7fe8f419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 -- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8f41022e0 msgr2=0x7fe8f419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:16.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8f41022e0 0x7fe8f419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 -- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f4102c90 msgr2=0x7fe8f419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f4102c90 0x7fe8f419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 -- 172.21.15.12:0/1584914186 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e8009520 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f1d9b700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe8f4102c90 0x7fe8f419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f2d9d700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f41073c0 0x7fe8f4197a90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00b070 tx=0x7fe8ec007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00e040 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ec009d00 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe8ec01d470 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.853+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0052a0 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.854+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f41a47a0 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.854+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8f41118b0 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.855+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7fe8ec004a50 con 0x7fe8f41073c0 2024-03-26T04:46:16.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.855+0000 7fe8e37fe700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe8dc06dac0 0x7fe8dc06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7fe8ec08f6a0 con 0x7fe8f41073c0 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8f259c700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe8dc06dac0 0x7fe8dc06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8e37fe700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] conn(0x7fe8dc0779d0 0x7fe8dc079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 --> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8dc07a540 con 0x7fe8dc0779d0 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8ec08faf0 con 0x7fe8f41073c0 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8f259c700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe8dc06dac0 0x7fe8dc06ff80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4198bd0 tx=0x7fe8e400b410 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8f1d9b700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] conn(0x7fe8dc0779d0 0x7fe8dc079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:16.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.856+0000 7fe8f1d9b700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] conn(0x7fe8dc0779d0 0x7fe8dc079e70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.857+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== osd.7 v2:172.21.15.186:6824/1742409046 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe8dc07a540 con 0x7fe8dc0779d0 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.874+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 --> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe8f40feae0 con 0x7fe8dc0779d0 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.874+0000 7fe8e37fe700 1 -- 172.21.15.12:0/1584914186 <== osd.7 v2:172.21.15.186:6824/1742409046 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe8f40feae0 con 0x7fe8dc0779d0 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] conn(0x7fe8dc0779d0 msgr2=0x7fe8dc079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.186:6824/1742409046,v1:172.21.15.186:6825/1742409046] conn(0x7fe8dc0779d0 0x7fe8dc079e70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe8dc06dac0 msgr2=0x7fe8dc06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe8dc06dac0 0x7fe8dc06ff80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4198bd0 tx=0x7fe8e400b410 comp rx=0 tx=0).stop 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f41073c0 msgr2=0x7fe8f4197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 --2- 172.21.15.12:0/1584914186 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe8f41073c0 0x7fe8f4197a90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00b070 tx=0x7fe8ec007930 comp rx=0 tx=0).stop 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f2d9d700 1 -- 172.21.15.12:0/1584914186 reap_dead start 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 shutdown_connections 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 >> 172.21.15.12:0/1584914186 conn(0x7fe8f40fdb50 msgr2=0x7fe8f40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:16.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 shutdown_connections 2024-03-26T04:46:16.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:15.875+0000 7fe8f8eac700 1 -- 172.21.15.12:0/1584914186 wait complete. 2024-03-26T04:46:16.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:16 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:16.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:16 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:16.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:46:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:16.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:16 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:16.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:16 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:16.935 INFO:teuthology.orchestra.run.smithi012.stdout:231928234000 2024-03-26T04:46:16.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T04:46:16.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:16 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:16.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:16 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 -- 172.21.15.12:0/1102030472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f701080d0 msgr2=0x7f6f701084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/1102030472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f701080d0 0x7f6f701084b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6f600099c0 tx=0x7f6f60009cd0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 -- 172.21.15.12:0/1102030472 shutdown_connections 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/1102030472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f701080d0 0x7f6f701084b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/1102030472 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 0x7f6f7010ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/1102030472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f70069d30 0x7f6f7006a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 -- 172.21.15.12:0/1102030472 >> 172.21.15.12:0/1102030472 conn(0x7f6f70077eb0 msgr2=0x7f6f700782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 -- 172.21.15.12:0/1102030472 shutdown_connections 2024-03-26T04:46:17.085 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.084+0000 7f6f74cb1700 1 -- 172.21.15.12:0/1102030472 wait complete. 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.085+0000 7f6f74cb1700 1 Processor -- start 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.085+0000 7f6f74cb1700 1 -- start start 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f74cb1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f70069d30 0x7f6f7019d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f74cb1700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 0x7f6f7019dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f74cb1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f701080d0 0x7f6f70197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f74cb1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70197fc0 con 0x7f6f70069d30 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f74cb1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70198100 con 0x7f6f701080d0 2024-03-26T04:46:17.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f74cb1700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f701982c0 con 0x7f6f7006a6d0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 0x7f6f7019dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6e59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f70069d30 0x7f6f7019d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 0x7f6f7019dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48696/0 (socket says 172.21.15.12:48696) 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6e59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f70069d30 0x7f6f7019d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43542/0 (socket says 172.21.15.12:43542) 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 -- 172.21.15.12:0/2761298972 learned_addr learned my addr 172.21.15.12:0/2761298972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6ed9d700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f701080d0 0x7f6f70197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 -- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f701080d0 msgr2=0x7f6f70197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f701080d0 0x7f6f70197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 -- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f70069d30 msgr2=0x7f6f7019d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6f70069d30 0x7f6f7019d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.086+0000 7f6f6dd9b700 1 -- 172.21.15.12:0/2761298972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f600095f0 con 0x7f6f7006a6d0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.087+0000 7f6f6dd9b700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 0x7f6f7019dce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400e9a0 tx=0x7f6f6400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.087+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64009790 con 0x7f6f7006a6d0 2024-03-26T04:46:17.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.087+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6f6400ce90 con 0x7f6f7006a6d0 2024-03-26T04:46:17.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.087+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64018420 con 0x7f6f7006a6d0 2024-03-26T04:46:17.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.087+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f701985a0 con 0x7f6f7006a6d0 2024-03-26T04:46:17.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.087+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f701a4840 con 0x7f6f7006a6d0 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.088+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f701117e0 con 0x7f6f7006a6d0 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.089+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f6f6400ca10 con 0x7f6f7006a6d0 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.089+0000 7f6f5f7fe700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f6f5806dac0 0x7f6f5806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.089+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f6f64014070 con 0x7f6f7006a6d0 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.089+0000 7f6f6e59c700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f6f5806dac0 0x7f6f5806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.090+0000 7f6f5f7fe700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] conn(0x7f6f580779d0 0x7f6f58079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.090+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 --> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f5807a540 con 0x7f6f580779d0 2024-03-26T04:46:17.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.090+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f640922c0 con 0x7f6f7006a6d0 2024-03-26T04:46:17.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.090+0000 7f6f6e59c700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f6f5806dac0 0x7f6f5806ff80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000b580 tx=0x7f6f60005fb0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:17.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.090+0000 7f6f6ed9d700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] conn(0x7f6f580779d0 0x7f6f58079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:17.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.090+0000 7f6f6ed9d700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] conn(0x7f6f580779d0 0x7f6f58079e70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:17.091 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.091+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== osd.6 v2:172.21.15.186:6816/2797487459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6f5807a540 con 0x7f6f580779d0 2024-03-26T04:46:17.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.108+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 --> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f701075f0 con 0x7f6f580779d0 2024-03-26T04:46:17.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f5f7fe700 1 -- 172.21.15.12:0/2761298972 <== osd.6 v2:172.21.15.186:6816/2797487459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6f701075f0 con 0x7f6f580779d0 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] conn(0x7f6f580779d0 msgr2=0x7f6f58079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.186:6816/2797487459,v1:172.21.15.186:6817/2797487459] conn(0x7f6f580779d0 0x7f6f58079e70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f6f5806dac0 msgr2=0x7f6f5806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f6f5806dac0 0x7f6f5806ff80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000b580 tx=0x7f6f60005fb0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 msgr2=0x7f6f7019dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f74cb1700 1 --2- 172.21.15.12:0/2761298972 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f6f7006a6d0 0x7f6f7019dce0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400e9a0 tx=0x7f6f6400ecb0 comp rx=0 tx=0).stop 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.109+0000 7f6f6ed9d700 1 -- 172.21.15.12:0/2761298972 reap_dead start 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.110+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 shutdown_connections 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.110+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 >> 172.21.15.12:0/2761298972 conn(0x7f6f70077eb0 msgr2=0x7f6f70105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.110+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 shutdown_connections 2024-03-26T04:46:17.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:17.110+0000 7f6f74cb1700 1 -- 172.21.15.12:0/2761298972 wait complete. 2024-03-26T04:46:17.505 INFO:teuthology.orchestra.run.smithi012.stdout:107374182436 2024-03-26T04:46:17.506 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T04:46:17.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:17 smithi186 ceph-mon[33438]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:17.964 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:17 smithi012 ceph-mon[41141]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:17.965 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:17 smithi012 ceph-mon[29863]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:17.965 INFO:teuthology.orchestra.run.smithi012.stdout:55834574893 2024-03-26T04:46:17.966 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T04:46:18.489 INFO:teuthology.orchestra.run.smithi012.stdout:197568495637 2024-03-26T04:46:18.490 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 -- 172.21.15.12:0/1210360967 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 msgr2=0x7f741810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 --2- 172.21.15.12:0/1210360967 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741810de00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f741000b0a0 tx=0x7f741000b3b0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 -- 172.21.15.12:0/1210360967 shutdown_connections 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 --2- 172.21.15.12:0/1210360967 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7418106fc0 0x7f74181073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 --2- 172.21.15.12:0/1210360967 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741810de00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 --2- 172.21.15.12:0/1210360967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7418100f80 0x7f74181013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 -- 172.21.15.12:0/1210360967 >> 172.21.15.12:0/1210360967 conn(0x7f7418077eb0 msgr2=0x7f74180782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 -- 172.21.15.12:0/1210360967 shutdown_connections 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 -- 172.21.15.12:0/1210360967 wait complete. 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 Processor -- start 2024-03-26T04:46:18.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.604+0000 7f741cac0700 1 -- start start 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7418100f80 0x7f741819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7418106fc0 0x7f7418197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181980f0 con 0x7f7418106fc0 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418198230 con 0x7f7418100f80 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74181983f0 con 0x7f7418101920 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7416d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7418106fc0 0x7f7418197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:48642/0 (socket says 172.21.15.12:48642) 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 -- 172.21.15.12:0/1858016993 learned_addr learned my addr 172.21.15.12:0/1858016993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741659c700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7418100f80 0x7f741819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 -- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7418100f80 msgr2=0x7f741819d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7418100f80 0x7f741819d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 -- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7418106fc0 msgr2=0x7f7418197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7418106fc0 0x7f7418197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 -- 172.21.15.12:0/1858016993 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7410009d00 con 0x7f7418101920 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741659c700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7418100f80 0x7f741819d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f7415d9b700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741819de10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d950 tx=0x7f740c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c009940 con 0x7f7418101920 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74181986d0 con 0x7f7418101920 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.605+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f740c010460 con 0x7f7418101920 2024-03-26T04:46:18.606 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.606+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c00f620 con 0x7f7418101920 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.606+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74181a4970 con 0x7f7418101920 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.606+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7418111940 con 0x7f7418101920 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.607+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f740c009aa0 con 0x7f7418101920 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.607+0000 7f74077fe700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f740006dac0 0x7f740006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.607+0000 7f741659c700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f740006dac0 0x7f740006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.607+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f740c08dbb0 con 0x7f7418101920 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.608+0000 7f74077fe700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] conn(0x7f7400077a50 0x7f7400079ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.608+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 --> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f740007a5c0 con 0x7f7400077a50 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.608+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f740c08e000 con 0x7f7418101920 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.608+0000 7f741659c700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f740006dac0 0x7f740006ff80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7418198bd0 tx=0x7f7408009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.608+0000 7f7416d9d700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] conn(0x7f7400077a50 0x7f7400079ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.608+0000 7f7416d9d700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] conn(0x7f7400077a50 0x7f7400079ef0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.609+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== osd.4 v2:172.21.15.186:6800/3595955502 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f740007a5c0 con 0x7f7400077a50 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.625+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 --> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7418062710 con 0x7f7400077a50 2024-03-26T04:46:18.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.626+0000 7f74077fe700 1 -- 172.21.15.12:0/1858016993 <== osd.4 v2:172.21.15.186:6800/3595955502 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7418062710 con 0x7f7400077a50 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] conn(0x7f7400077a50 msgr2=0x7f7400079ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.186:6800/3595955502,v1:172.21.15.186:6801/3595955502] conn(0x7f7400077a50 0x7f7400079ef0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f740006dac0 msgr2=0x7f740006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f740006dac0 0x7f740006ff80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7418198bd0 tx=0x7f7408009260 comp rx=0 tx=0).stop 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 msgr2=0x7f741819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 --2- 172.21.15.12:0/1858016993 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7418101920 0x7f741819de10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d950 tx=0x7f740c00dc60 comp rx=0 tx=0).stop 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f7416d9d700 1 -- 172.21.15.12:0/1858016993 reap_dead start 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 shutdown_connections 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 >> 172.21.15.12:0/1858016993 conn(0x7f7418077eb0 msgr2=0x7f74180ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 shutdown_connections 2024-03-26T04:46:18.608 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:14.627+0000 7f741cac0700 1 -- 172.21.15.12:0/1858016993 wait complete. 2024-03-26T04:46:19.202 INFO:teuthology.orchestra.run.smithi012.stdout:128849018910 2024-03-26T04:46:19.203 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T04:46:19.242 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:19.340 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:19.577 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:19.580 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:19.581 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:19.588 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:20.135 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.136 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi012 to 3761M 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[41141]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.137 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi012 to 3761M 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:46:20.138 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:19 smithi012 ceph-mon[29863]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi012 to 3761M 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: Updating smithi012:/etc/ceph/ceph.conf 2024-03-26T04:46:20.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:19 smithi186 ceph-mon[33438]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.005 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.006 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: Updating smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: Updating smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:22.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:21 smithi012 ceph-mon[29863]: Deploying daemon alertmanager.a on smithi012 2024-03-26T04:46:22.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:21 smithi012 ceph-mon[29863]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:22.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:21 smithi012 ceph-mon[41141]: Deploying daemon alertmanager.a on smithi012 2024-03-26T04:46:22.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:21 smithi012 ceph-mon[41141]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:21 smithi186 ceph-mon[33438]: Deploying daemon alertmanager.a on smithi012 2024-03-26T04:46:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:21 smithi186 ceph-mon[33438]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:22.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:46:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:46:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:22 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:23.226 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:22 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:23.226 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:22 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:24.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:23 smithi186 ceph-mon[33438]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:24.221 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:24.299 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:23 smithi012 ceph-mon[29863]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:24.299 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:23 smithi012 ceph-mon[41141]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.676+0000 7fe79f523700 1 -- 172.21.15.12:0/1669438645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798100f80 msgr2=0x7fe7981013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.676+0000 7fe79f523700 1 --2- 172.21.15.12:0/1669438645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798100f80 0x7fe7981013e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe7880098f0 tx=0x7fe788009c00 comp rx=0 tx=0).stop 2024-03-26T04:46:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 -- 172.21.15.12:0/1669438645 shutdown_connections 2024-03-26T04:46:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 --2- 172.21.15.12:0/1669438645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798106fc0 0x7fe7981073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 --2- 172.21.15.12:0/1669438645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe798101920 0x7fe79810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 --2- 172.21.15.12:0/1669438645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798100f80 0x7fe7981013e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 -- 172.21.15.12:0/1669438645 >> 172.21.15.12:0/1669438645 conn(0x7fe798077eb0 msgr2=0x7fe7980782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 -- 172.21.15.12:0/1669438645 shutdown_connections 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 -- 172.21.15.12:0/1669438645 wait complete. 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 Processor -- start 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.677+0000 7fe79f523700 1 -- start start 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe798100f80 0x7fe7981967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 0x7fe798196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798106fc0 0x7fe798197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a5f20 con 0x7fe798101920 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a6090 con 0x7fe798100f80 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7981a6250 con 0x7fe798106fc0 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 0x7fe798196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:24.778 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 0x7fe798196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47996/0 (socket says 172.21.15.12:47996) 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 -- 172.21.15.12:0/4141220693 learned_addr learned my addr 172.21.15.12:0/4141220693 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79dac0700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798106fc0 0x7fe798197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79d2bf700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe798100f80 0x7fe7981967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 -- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798106fc0 msgr2=0x7fe798197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798106fc0 0x7fe798197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 -- 172.21.15.12:0/4141220693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe798100f80 msgr2=0x7fe7981967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe798100f80 0x7fe7981967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 -- 172.21.15.12:0/4141220693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940095f0 con 0x7fe798101920 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79dac0700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798106fc0 0x7fe798197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79cabe700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 0x7fe798196d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe788011040 tx=0x7fe78800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78801c070 con 0x7fe798101920 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe78800bae0 con 0x7fe798101920 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe788009520 con 0x7fe798101920 2024-03-26T04:46:24.779 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.678+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788004930 con 0x7fe798101920 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.679+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7981a6750 con 0x7fe798101920 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.679+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7981094c0 con 0x7fe798101920 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.680+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7fe788020070 con 0x7fe798101920 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.680+0000 7fe78e7fc700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe78406dac0 0x7fe78406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.681+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7fe78808e640 con 0x7fe798101920 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.681+0000 7fe79d2bf700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe78406dac0 0x7fe78406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.681+0000 7fe79d2bf700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe78406dac0 0x7fe78406ff80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe794005fd0 tx=0x7fe794009310 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:24.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.685+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe788057b50 con 0x7fe798101920 2024-03-26T04:46:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.895+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe79806a320 con 0x7fe798101920 2024-03-26T04:46:24.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.896+0000 7fe78e7fc700 1 -- 172.21.15.12:0/4141220693 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe78805b210 con 0x7fe798101920 2024-03-26T04:46:24.896 INFO:teuthology.orchestra.run.smithi012.stdout:55834574895 2024-03-26T04:46:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe78406dac0 msgr2=0x7fe78406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe78406dac0 0x7fe78406ff80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe794005fd0 tx=0x7fe794009310 comp rx=0 tx=0).stop 2024-03-26T04:46:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 msgr2=0x7fe798196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 0x7fe798196d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe788011040 tx=0x7fe78800fa00 comp rx=0 tx=0).stop 2024-03-26T04:46:24.898 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 shutdown_connections 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fe78406dac0 0x7fe78406ff80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fe798106fc0 0x7fe798197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe798100f80 0x7fe7981967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 --2- 172.21.15.12:0/4141220693 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe798101920 0x7fe798196d20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 >> 172.21.15.12:0/4141220693 conn(0x7fe798077eb0 msgr2=0x7fe7980ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 shutdown_connections 2024-03-26T04:46:24.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:24.898+0000 7fe79f523700 1 -- 172.21.15.12:0/4141220693 wait complete. 2024-03-26T04:46:25.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4141220693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:46:25.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4141220693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:46:25.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4141220693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:46:26.074 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:25 smithi012 ceph-mon[29863]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:26.075 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:25 smithi012 ceph-mon[41141]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:26.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:25 smithi186 ceph-mon[33438]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:26.422 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574895 for osd.1 2024-03-26T04:46:26.422 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:26.996 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:46:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:27.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 -- 172.21.15.12:0/4104836090 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f8069d30 msgr2=0x7f42f806a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:27.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 --2- 172.21.15.12:0/4104836090 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f8069d30 0x7f42f806a190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0099c0 tx=0x7f42ec009cd0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 -- 172.21.15.12:0/4104836090 shutdown_connections 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 --2- 172.21.15.12:0/4104836090 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f806a6d0 0x7f42f810ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 --2- 172.21.15.12:0/4104836090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f81080d0 0x7f42f81084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 --2- 172.21.15.12:0/4104836090 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f8069d30 0x7f42f806a190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 -- 172.21.15.12:0/4104836090 >> 172.21.15.12:0/4104836090 conn(0x7f42f8077eb0 msgr2=0x7f42f80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 -- 172.21.15.12:0/4104836090 shutdown_connections 2024-03-26T04:46:27.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.018+0000 7f42fcaef700 1 -- 172.21.15.12:0/4104836090 wait complete. 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 Processor -- start 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 -- start start 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8069d30 0x7f42f8101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f806a6d0 0x7f42f8101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 0x7f42f8101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81a5f30 con 0x7f42f81080d0 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81a60a0 con 0x7f42f8069d30 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42fcaef700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81a6260 con 0x7f42f806a6d0 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42f6d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 0x7f42f8101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42f5d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f806a6d0 0x7f42f8101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42f6d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 0x7f42f8101ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48012/0 (socket says 172.21.15.12:48012) 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42f6d9d700 1 -- 172.21.15.12:0/2213179181 learned_addr learned my addr 172.21.15.12:0/2213179181 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.019+0000 7f42f659c700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8069d30 0x7f42f8101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:27.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f5d9b700 1 -- 172.21.15.12:0/2213179181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8069d30 msgr2=0x7f42f8101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f5d9b700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8069d30 0x7f42f8101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f5d9b700 1 -- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 msgr2=0x7f42f8101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f5d9b700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 0x7f42f8101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f5d9b700 1 -- 172.21.15.12:0/2213179181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0009c40 con 0x7f42f806a6d0 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f6d9d700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 0x7f42f8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42f5d9b700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f806a6d0 0x7f42f8101590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0098e0 tx=0x7f42ec00b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec01d070 con 0x7f42f806a6d0 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f42ec00bdc0 con 0x7f42f806a6d0 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00f8d0 con 0x7f42f806a6d0 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec0095f0 con 0x7f42f806a6d0 2024-03-26T04:46:27.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.020+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f81a67c0 con 0x7f42f806a6d0 2024-03-26T04:46:27.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.021+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f8068d40 con 0x7f42f806a6d0 2024-03-26T04:46:27.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.022+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f42ec003c90 con 0x7f42f806a6d0 2024-03-26T04:46:27.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.022+0000 7f42e77fe700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f42e006dac0 0x7f42e006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:27.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.022+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f42ec08e540 con 0x7f42f806a6d0 2024-03-26T04:46:27.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.022+0000 7f42f659c700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f42e006dac0 0x7f42e006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:27.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.023+0000 7f42f659c700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f42e006dac0 0x7f42e006ff80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42e8005950 tx=0x7f42e800b410 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.027+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f42ec0579b0 con 0x7f42f806a6d0 2024-03-26T04:46:27.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.243+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f42f81115d0 con 0x7f42f806a6d0 2024-03-26T04:46:27.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.243+0000 7f42e77fe700 1 -- 172.21.15.12:0/2213179181 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42ec05b070 con 0x7f42f806a6d0 2024-03-26T04:46:27.244 INFO:teuthology.orchestra.run.smithi012.stdout:77309411371 2024-03-26T04:46:27.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.245+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f42e006dac0 msgr2=0x7f42e006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:27.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.245+0000 7f42fcaef700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f42e006dac0 0x7f42e006ff80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f42e8005950 tx=0x7f42e800b410 comp rx=0 tx=0).stop 2024-03-26T04:46:27.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.245+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f806a6d0 msgr2=0x7f42f8101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:27.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.245+0000 7f42fcaef700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f806a6d0 0x7f42f8101590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0098e0 tx=0x7f42ec00b9c0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 shutdown_connections 2024-03-26T04:46:27.246 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f42e006dac0 0x7f42e006ff80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f42f81080d0 0x7f42f8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8069d30 0x7f42f8101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 --2- 172.21.15.12:0/2213179181 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f42f806a6d0 0x7f42f8101590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:27.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 >> 172.21.15.12:0/2213179181 conn(0x7f42f8077eb0 msgr2=0x7f42f8108df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:27.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 shutdown_connections 2024-03-26T04:46:27.247 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:27.246+0000 7f42fcaef700 1 -- 172.21.15.12:0/2213179181 wait complete. 2024-03-26T04:46:28.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:27 smithi012 ceph-mon[29863]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:28.069 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2213179181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:46:28.069 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:27 smithi012 ceph-mon[41141]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:28.069 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2213179181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:46:28.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:27 smithi186 ceph-mon[33438]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:28.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2213179181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:46:29.043 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:28 smithi012 ceph-mon[29863]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T04:46:29.043 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:28 smithi012 ceph-mon[41141]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T04:46:29.044 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2024-03-26T04:46:29.044 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:29.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:28 smithi186 ceph-mon[33438]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 -- 172.21.15.12:0/3327060630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 msgr2=0x7f9ac4102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3327060630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac4102550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009920 tx=0x7f9ac0009c30 comp rx=0 tx=0).stop 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 -- 172.21.15.12:0/3327060630 shutdown_connections 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3327060630 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac4102a90 0x7f9ac410eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3327060630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 0x7f9ac4108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3327060630 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac4102550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 -- 172.21.15.12:0/3327060630 >> 172.21.15.12:0/3327060630 conn(0x7f9ac40791b0 msgr2=0x7f9ac40795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 -- 172.21.15.12:0/3327060630 shutdown_connections 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.606+0000 7f9acc70d700 1 -- 172.21.15.12:0/3327060630 wait complete. 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 Processor -- start 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 -- start start 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac419d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:29.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac4102a90 0x7f9ac419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 0x7f9ac4197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4197fc0 con 0x7f9ac41020d0 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4198100 con 0x7f9ac41080b0 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acc70d700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac41982c0 con 0x7f9ac4102a90 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac419d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9ac9ca8700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac4102a90 0x7f9ac419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac419d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56934/0 (socket says 172.21.15.12:56934) 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 -- 172.21.15.12:0/3827566603 learned_addr learned my addr 172.21.15.12:0/3827566603 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acacaa700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 0x7f9ac4197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 -- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac4102a90 msgr2=0x7f9ac419dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac4102a90 0x7f9ac419dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 -- 172.21.15.12:0/3827566603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 msgr2=0x7f9ac4197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 0x7f9ac4197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9aca4a9700 1 -- 172.21.15.12:0/3827566603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc009d00 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.607+0000 7f9acacaa700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 0x7f9ac4197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.608+0000 7f9aca4a9700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac419d7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c920 tx=0x7f9ab400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.608+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4007b00 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.608+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac0009520 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.608+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9ab400ce70 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.608+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4005300 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.608+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac41a4670 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.609+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac4068d40 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.609+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f9ab40054c0 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.610+0000 7f9abb7fe700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f9ab006dac0 0x7f9ab006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.610+0000 7f9ac9ca8700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f9ab006dac0 0x7f9ab006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.610+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f9ab408d460 con 0x7f9ac41020d0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.610+0000 7f9ac9ca8700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f9ab006dac0 0x7f9ab006ff80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4112230 tx=0x7f9ac0019040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:29.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.614+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9ab4056850 con 0x7f9ac41020d0 2024-03-26T04:46:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.823+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9ac4111560 con 0x7f9ac41020d0 2024-03-26T04:46:29.823 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.823+0000 7f9abb7fe700 1 -- 172.21.15.12:0/3827566603 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9ab4059f10 con 0x7f9ac41020d0 2024-03-26T04:46:29.824 INFO:teuthology.orchestra.run.smithi012.stdout:197568495639 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f9ab006dac0 msgr2=0x7f9ab006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f9ab006dac0 0x7f9ab006ff80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4112230 tx=0x7f9ac0019040 comp rx=0 tx=0).stop 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 msgr2=0x7f9ac419d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac419d7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9ab400c920 tx=0x7f9ab400cc30 comp rx=0 tx=0).stop 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 shutdown_connections 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f9ab006dac0 0x7f9ab006ff80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9ac41020d0 0x7f9ac419d7a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac41080b0 0x7f9ac4197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 --2- 172.21.15.12:0/3827566603 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f9ac4102a90 0x7f9ac419dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 >> 172.21.15.12:0/3827566603 conn(0x7f9ac40791b0 msgr2=0x7f9ac4108e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:29.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 shutdown_connections 2024-03-26T04:46:29.827 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:29.825+0000 7f9acc70d700 1 -- 172.21.15.12:0/3827566603 wait complete. 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 -- 172.21.15.12:0/560241218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de406b190 msgr2=0x7f0de410eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 --2- 172.21.15.12:0/560241218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de406b190 0x7f0de410eed0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc009920 tx=0x7f0ddc009c30 comp rx=0 tx=0).stop 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 -- 172.21.15.12:0/560241218 shutdown_connections 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 --2- 172.21.15.12:0/560241218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de406b190 0x7f0de410eed0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 --2- 172.21.15.12:0/560241218 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de41080b0 0x7f0de4108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 --2- 172.21.15.12:0/560241218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0de406a7d0 0x7f0de406ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 -- 172.21.15.12:0/560241218 >> 172.21.15.12:0/560241218 conn(0x7f0de4077f90 msgr2=0x7f0de40783a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 -- 172.21.15.12:0/560241218 shutdown_connections 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 -- 172.21.15.12:0/560241218 wait complete. 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.013+0000 7f0de9688700 1 Processor -- start 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 -- start start 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0de406a7d0 0x7f0de419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 0x7f0de419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de41080b0 0x7f0de41978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4197f00 con 0x7f0de41080b0 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4198040 con 0x7f0de406a7d0 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4198200 con 0x7f0de406b190 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de27fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 0x7f0de419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de27fc700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 0x7f0de419dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44644/0 (socket says 172.21.15.12:44644) 2024-03-26T04:46:30.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de27fc700 1 -- 172.21.15.12:0/3269795558 learned_addr learned my addr 172.21.15.12:0/3269795558 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de41080b0 0x7f0de41978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de2ffd700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0de406a7d0 0x7f0de419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 -- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 msgr2=0x7f0de419dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 0x7f0de419dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 -- 172.21.15.12:0/3269795558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0de406a7d0 msgr2=0x7f0de419d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0de406a7d0 0x7f0de419d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 -- 172.21.15.12:0/3269795558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4009d00 con 0x7f0de41080b0 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de27fc700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 0x7f0de419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de37fe700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de41080b0 0x7f0de41978a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc005e90 tx=0x7f0ddc00fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc01c070 con 0x7f0de41080b0 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0ddc00b9e0 con 0x7f0de41080b0 2024-03-26T04:46:30.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.014+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc009520 con 0x7f0de41080b0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.015+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc0046d0 con 0x7f0de41080b0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.015+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de41986b0 con 0x7f0de41080b0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.015+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de4068d40 con 0x7f0de41080b0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.016+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f0ddc020070 con 0x7f0de41080b0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.016+0000 7f0ddbfff700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f0dd006dac0 0x7f0dd006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.017+0000 7f0de2ffd700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f0dd006dac0 0x7f0dd006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.017+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f0ddc08e4e0 con 0x7f0de41080b0 2024-03-26T04:46:30.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.017+0000 7f0de2ffd700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f0dd006dac0 0x7f0dd006ff80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0de41989e0 tx=0x7f0dd400b040 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:30.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.021+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f0ddc0578d0 con 0x7f0de41080b0 2024-03-26T04:46:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3827566603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:46:30.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.238+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0de41115b0 con 0x7f0de41080b0 2024-03-26T04:46:30.239 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3827566603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:46:30.239 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3827566603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:46:30.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.238+0000 7f0ddbfff700 1 -- 172.21.15.12:0/3269795558 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0ddc05af90 con 0x7f0de41080b0 2024-03-26T04:46:30.240 INFO:teuthology.orchestra.run.smithi012.stdout:128849018913 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f0dd006dac0 msgr2=0x7f0dd006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f0dd006dac0 0x7f0dd006ff80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0de41989e0 tx=0x7f0dd400b040 comp rx=0 tx=0).stop 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de41080b0 msgr2=0x7f0de41978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de41080b0 0x7f0de41978a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc005e90 tx=0x7f0ddc00fa20 comp rx=0 tx=0).stop 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 shutdown_connections 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f0dd006dac0 0x7f0dd006ff80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0de41080b0 0x7f0de41978a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0de406a7d0 0x7f0de419d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 --2- 172.21.15.12:0/3269795558 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f0de406b190 0x7f0de419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.240+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 >> 172.21.15.12:0/3269795558 conn(0x7f0de4077f90 msgr2=0x7f0de4105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.241+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 shutdown_connections 2024-03-26T04:46:30.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:30.241+0000 7f0de9688700 1 -- 172.21.15.12:0/3269795558 wait complete. 2024-03-26T04:46:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:30 smithi186 ceph-mon[33438]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:31.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:30 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:31.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3269795558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:46:31.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:30 smithi012 ceph-mon[29863]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:31.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:30 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:31.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3269795558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:46:31.354 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:30 smithi012 ceph-mon[41141]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:31.354 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:30 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:31.354 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3269795558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:46:31.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 -- 172.21.15.12:0/1480339990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c06a980 msgr2=0x7fdc6c06ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 --2- 172.21.15.12:0/1480339990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c06a980 0x7fdc6c06ad60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009990 tx=0x7fdc68009ca0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 -- 172.21.15.12:0/1480339990 shutdown_connections 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 --2- 172.21.15.12:0/1480339990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c1073f0 0x7fdc6c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 --2- 172.21.15.12:0/1480339990 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c06a980 0x7fdc6c06ad60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 --2- 172.21.15.12:0/1480339990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 -- 172.21.15.12:0/1480339990 >> 172.21.15.12:0/1480339990 conn(0x7fdc6c077ef0 msgr2=0x7fdc6c078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 -- 172.21.15.12:0/1480339990 shutdown_connections 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 -- 172.21.15.12:0/1480339990 wait complete. 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.617+0000 7fdc735fe700 1 Processor -- start 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 -- start start 2024-03-26T04:46:31.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c06a980 0x7fdc6c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c1073f0 0x7fdc6c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c198040 con 0x7fdc6c06b330 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c198180 con 0x7fdc6c06a980 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc735fe700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c198340 con 0x7fdc6c1073f0 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c19dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56974/0 (socket says 172.21.15.12:56974) 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc71b9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c1073f0 0x7fdc6c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 -- 172.21.15.12:0/2747218023 learned_addr learned my addr 172.21.15.12:0/2747218023 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc7139a700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c06a980 0x7fdc6c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 -- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c1073f0 msgr2=0x7fdc6c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c1073f0 0x7fdc6c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 -- 172.21.15.12:0/2747218023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c06a980 msgr2=0x7fdc6c19d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c06a980 0x7fdc6c19d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 -- 172.21.15.12:0/2747218023 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c009520 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc71b9b700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c1073f0 0x7fdc6c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.618+0000 7fdc70b99700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c19dd60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009fd0 tx=0x7fdc5c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.619+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c00cbd0 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.619+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc680095f0 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.619+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdc5c00eea0 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.619+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c003b70 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.619+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c100f70 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.620+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c1037a0 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.620+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7fdc5c003d30 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.620+0000 7fdc627fc700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fdc5806dac0 0x7fdc5806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.621+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7fdc5c014070 con 0x7fdc6c06b330 2024-03-26T04:46:31.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.621+0000 7fdc7139a700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fdc5806dac0 0x7fdc5806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:31.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.621+0000 7fdc7139a700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fdc5806dac0 0x7fdc5806ff80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c198b20 tx=0x7fdc6800b260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:31.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.625+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdc5c056f10 con 0x7fdc6c06b330 2024-03-26T04:46:31.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.833+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdc6c068d40 con 0x7fdc6c06b330 2024-03-26T04:46:31.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.833+0000 7fdc627fc700 1 -- 172.21.15.12:0/2747218023 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdc5c05a5d0 con 0x7fdc6c06b330 2024-03-26T04:46:31.834 INFO:teuthology.orchestra.run.smithi012.stdout:231928234003 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.835+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fdc5806dac0 msgr2=0x7fdc5806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.835+0000 7fdc735fe700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fdc5806dac0 0x7fdc5806ff80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c198b20 tx=0x7fdc6800b260 comp rx=0 tx=0).stop 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.835+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 msgr2=0x7fdc6c19dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.835+0000 7fdc735fe700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c19dd60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009fd0 tx=0x7fdc5c00ed10 comp rx=0 tx=0).stop 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 shutdown_connections 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fdc5806dac0 0x7fdc5806ff80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fdc6c1073f0 0x7fdc6c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c06a980 0x7fdc6c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 --2- 172.21.15.12:0/2747218023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc6c06b330 0x7fdc6c19dd60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 >> 172.21.15.12:0/2747218023 conn(0x7fdc6c077ef0 msgr2=0x7fdc6c0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 shutdown_connections 2024-03-26T04:46:31.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:31.836+0000 7fdc735fe700 1 -- 172.21.15.12:0/2747218023 wait complete. 2024-03-26T04:46:32.182 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-03-26T04:46:32.182 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:32.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2747218023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:46:32.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:31 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:32.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:46:31 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:46:32.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2747218023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:46:32.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:31 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:32.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2747218023' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:46:32.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:31 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:33.265 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:32 smithi012 ceph-mon[29863]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:33.265 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:32 smithi012 ceph-mon[41141]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:33.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:32 smithi186 ceph-mon[33438]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:33.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234003 for osd.7 2024-03-26T04:46:33.889 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:34.017 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495637 got 197568495639 for osd.6 2024-03-26T04:46:34.017 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:35.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:35 smithi012 ceph-mon[29863]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:35.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2300491975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:46:35.357 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:35 smithi012 ceph-mon[41141]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:35.357 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2300491975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 -- 172.21.15.12:0/1237907459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881080b0 msgr2=0x7f5588108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 --2- 172.21.15.12:0/1237907459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881080b0 0x7f5588108490 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f557800b0d0 tx=0x7f557800b3e0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 -- 172.21.15.12:0/1237907459 shutdown_connections 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 --2- 172.21.15.12:0/1237907459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881080b0 0x7f5588108490 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 --2- 172.21.15.12:0/1237907459 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 0x7f558810eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 --2- 172.21.15.12:0/1237907459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55881020d0 0x7f5588102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 -- 172.21.15.12:0/1237907459 >> 172.21.15.12:0/1237907459 conn(0x7f55880791b0 msgr2=0x7f55880795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 -- 172.21.15.12:0/1237907459 shutdown_connections 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.532+0000 7f558f38a700 1 -- 172.21.15.12:0/1237907459 wait complete. 2024-03-26T04:46:35.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 Processor -- start 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 -- start start 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 0x7f558819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 0x7f558819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55881080b0 0x7f55881979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588198010 con 0x7f55881020d0 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5588198150 con 0x7f55881080b0 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558f38a700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5588198310 con 0x7f5588102a90 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558c925700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 0x7f558819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558d126700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 0x7f558819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558d126700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 0x7f558819d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56992/0 (socket says 172.21.15.12:56992) 2024-03-26T04:46:35.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558d126700 1 -- 172.21.15.12:0/2300491975 learned_addr learned my addr 172.21.15.12:0/2300491975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:35.442 INFO:teuthology.orchestra.run.smithi012.stdout:163208757278 2024-03-26T04:46:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.533+0000 7f558d927700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55881080b0 0x7f55881979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558d126700 1 -- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 msgr2=0x7f558819dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558d126700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 0x7f558819dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558d126700 1 -- 172.21.15.12:0/2300491975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55881080b0 msgr2=0x7f55881979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558d126700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55881080b0 0x7f55881979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558d126700 1 -- 172.21.15.12:0/2300491975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580009520 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558c925700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 0x7f558819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558d126700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 0x7f558819d7f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f557800ced0 tx=0x7f557800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557800e040 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578009d00 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5578003e80 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557801b470 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.534+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5588198810 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.535+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5588068d40 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.535+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f5578003a00 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.536+0000 7f55867fc700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f557006dac0 0x7f557006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.536+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f557808e920 con 0x7f55881020d0 2024-03-26T04:46:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.536+0000 7f558c925700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f557006dac0 0x7f557006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.536+0000 7f558c925700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f557006dac0 0x7f557006ff80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f557c00ffd0 tx=0x7f557c00f820 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.540+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5578057d10 con 0x7f55881020d0 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.769+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5588111520 con 0x7f55881020d0 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.770+0000 7f55867fc700 1 -- 172.21.15.12:0/2300491975 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f557805b3d0 con 0x7f55881020d0 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f557006dac0 msgr2=0x7f557006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f557006dac0 0x7f557006ff80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f557c00ffd0 tx=0x7f557c00f820 comp rx=0 tx=0).stop 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 msgr2=0x7f558819d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 0x7f558819d7f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f557800ced0 tx=0x7f557800cf00 comp rx=0 tx=0).stop 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 shutdown_connections 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f557006dac0 0x7f557006ff80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f55881020d0 0x7f558819d7f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55881080b0 0x7f55881979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 --2- 172.21.15.12:0/2300491975 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5588102a90 0x7f558819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:35.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 >> 172.21.15.12:0/2300491975 conn(0x7f55880791b0 msgr2=0x7f5588105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:35.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 shutdown_connections 2024-03-26T04:46:35.445 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:34.772+0000 7f558f38a700 1 -- 172.21.15.12:0/2300491975 wait complete. 2024-03-26T04:46:35.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:35 smithi186 ceph-mon[33438]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:35.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2300491975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:46:35.977 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757274 got 163208757278 for osd.5 2024-03-26T04:46:35.978 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:36.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 -- 172.21.15.12:0/2335907197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 msgr2=0x7f7c8c105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 --2- 172.21.15.12:0/2335907197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c105bb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0098f0 tx=0x7f7c7c009c00 comp rx=0 tx=0).stop 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 -- 172.21.15.12:0/2335907197 shutdown_connections 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 --2- 172.21.15.12:0/2335907197 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 --2- 172.21.15.12:0/2335907197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c8c106290 0x7f7c8c10de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 --2- 172.21.15.12:0/2335907197 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c105bb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 -- 172.21.15.12:0/2335907197 >> 172.21.15.12:0/2335907197 conn(0x7f7c8c076fd0 msgr2=0x7f7c8c0773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 -- 172.21.15.12:0/2335907197 shutdown_connections 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.169+0000 7f7c90b28700 1 -- 172.21.15.12:0/2335907197 wait complete. 2024-03-26T04:46:36.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 Processor -- start 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 -- start start 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c8c106290 0x7f7c8c1036d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c1a5fc0 con 0x7f7c8c101240 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c1a6130 con 0x7f7c8c106290 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.170+0000 7f7c90b28700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c1a62f0 con 0x7f7c8c101bf0 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c8a59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c8a59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57014/0 (socket says 172.21.15.12:57014) 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c8a59c700 1 -- 172.21.15.12:0/1874328289 learned_addr learned my addr 172.21.15.12:0/1874328289 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:36.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:44700/0 (socket says 172.21.15.12:44700) 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c8ad9d700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c8c106290 0x7f7c8c1036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 -- 172.21.15.12:0/1874328289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c8c106290 msgr2=0x7f7c8c1036d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c8c106290 0x7f7c8c1036d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 -- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 msgr2=0x7f7c8c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 -- 172.21.15.12:0/1874328289 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740095f0 con 0x7f7c8c101bf0 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c8a59c700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c89d9b700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c103190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c000c00 tx=0x7f7c7c00fa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c01c070 con 0x7f7c8c101bf0 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c009520 con 0x7f7c8c101bf0 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7c7c00ba60 con 0x7f7c8c101bf0 2024-03-26T04:46:36.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c005720 con 0x7f7c8c101bf0 2024-03-26T04:46:36.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.171+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c1a67f0 con 0x7f7c8c101bf0 2024-03-26T04:46:36.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.172+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c068d40 con 0x7f7c8c101bf0 2024-03-26T04:46:36.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.173+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f7c7c020030 con 0x7f7c8c101bf0 2024-03-26T04:46:36.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.173+0000 7f7c837fe700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f7c7806dac0 0x7f7c7806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.173+0000 7f7c8a59c700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f7c7806dac0 0x7f7c7806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.173+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f7c7c08e6e0 con 0x7f7c8c101bf0 2024-03-26T04:46:36.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.174+0000 7f7c8a59c700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f7c7806dac0 0x7f7c7806ff80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c74005d10 tx=0x7f7c74009310 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:36.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.178+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7c7c057ad0 con 0x7f7c8c101bf0 2024-03-26T04:46:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.386+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7c8c110510 con 0x7f7c8c101bf0 2024-03-26T04:46:36.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.386+0000 7f7c837fe700 1 -- 172.21.15.12:0/1874328289 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7c7c05b190 con 0x7f7c8c101bf0 2024-03-26T04:46:36.387 INFO:teuthology.orchestra.run.smithi012.stdout:107374182441 2024-03-26T04:46:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f7c7806dac0 msgr2=0x7f7c7806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f7c7806dac0 0x7f7c7806ff80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c74005d10 tx=0x7f7c74009310 comp rx=0 tx=0).stop 2024-03-26T04:46:36.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 msgr2=0x7f7c8c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c103190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c000c00 tx=0x7f7c7c00fa20 comp rx=0 tx=0).stop 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 shutdown_connections 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f7c7806dac0 0x7f7c7806ff80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c8c101240 0x7f7c8c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c8c106290 0x7f7c8c1036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 --2- 172.21.15.12:0/1874328289 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f7c8c101bf0 0x7f7c8c103190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 >> 172.21.15.12:0/1874328289 conn(0x7f7c8c076fd0 msgr2=0x7f7c8c06a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 shutdown_connections 2024-03-26T04:46:36.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.389+0000 7f7c90b28700 1 -- 172.21.15.12:0/1874328289 wait complete. 2024-03-26T04:46:36.883 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:46:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:36.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 -- 172.21.15.12:0/329154228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c50106f70 msgr2=0x7f2c50069d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 --2- 172.21.15.12:0/329154228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c50106f70 0x7f2c50069d30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00b0a0 tx=0x7f2c4c00b3b0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 -- 172.21.15.12:0/329154228 shutdown_connections 2024-03-26T04:46:36.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 --2- 172.21.15.12:0/329154228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c50106f70 0x7f2c50069d30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 --2- 172.21.15.12:0/329154228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c50104a80 0x7f2c50104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 --2- 172.21.15.12:0/329154228 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c5006a270 0x7f2c5006a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 -- 172.21.15.12:0/329154228 >> 172.21.15.12:0/329154228 conn(0x7f2c50077eb0 msgr2=0x7f2c500782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 -- 172.21.15.12:0/329154228 shutdown_connections 2024-03-26T04:46:36.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 -- 172.21.15.12:0/329154228 wait complete. 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.989+0000 7f2c579aa700 1 Processor -- start 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 -- start start 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 0x7f2c5019e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c50104a80 0x7f2c5019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c50106f70 0x7f2c501a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c501a3220 con 0x7f2c5006a270 2024-03-26T04:46:36.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c501a3390 con 0x7f2c50104a80 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c579aa700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c501a3550 con 0x7f2c50106f70 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55746700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 0x7f2c5019e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55746700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 0x7f2c5019e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57028/0 (socket says 172.21.15.12:57028) 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55746700 1 -- 172.21.15.12:0/3697474675 learned_addr learned my addr 172.21.15.12:0/3697474675 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c50106f70 0x7f2c501a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c54f45700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c50104a80 0x7f2c5019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 -- 172.21.15.12:0/3697474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c50104a80 msgr2=0x7f2c5019e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c50104a80 0x7f2c5019e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 -- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 msgr2=0x7f2c5019e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 0x7f2c5019e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 -- 172.21.15.12:0/3697474675 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c40009520 con 0x7f2c50106f70 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55746700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 0x7f2c5019e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.990+0000 7f2c55f47700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c50106f70 0x7f2c501a2c10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800c7a0 tx=0x7f2c4800cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:36.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.991+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48004a30 con 0x7f2c50106f70 2024-03-26T04:46:36.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.991+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2c480054a0 con 0x7f2c50106f70 2024-03-26T04:46:36.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.991+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c009d00 con 0x7f2c50106f70 2024-03-26T04:46:36.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.991+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800ea60 con 0x7f2c50106f70 2024-03-26T04:46:36.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.991+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c501aaec0 con 0x7f2c50106f70 2024-03-26T04:46:36.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.992+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c50068d40 con 0x7f2c50106f70 2024-03-26T04:46:36.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.992+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f2c4800eca0 con 0x7f2c50106f70 2024-03-26T04:46:36.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.993+0000 7f2c467fc700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2c3c06dac0 0x7f2c3c06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:36.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.993+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f2c4808d320 con 0x7f2c50106f70 2024-03-26T04:46:36.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.993+0000 7f2c55746700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2c3c06dac0 0x7f2c3c06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:36.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.993+0000 7f2c55746700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2c3c06dac0 0x7f2c3c06ff80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009fd0 tx=0x7f2c4c006210 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:36.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:36.997+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2c480566f0 con 0x7f2c50106f70 2024-03-26T04:46:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.205+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2c50117bd0 con 0x7f2c50106f70 2024-03-26T04:46:37.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.205+0000 7f2c467fc700 1 -- 172.21.15.12:0/3697474675 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2c48059db0 con 0x7f2c50106f70 2024-03-26T04:46:37.206 INFO:teuthology.orchestra.run.smithi012.stdout:34359738424 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2c3c06dac0 msgr2=0x7f2c3c06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2c3c06dac0 0x7f2c3c06ff80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009fd0 tx=0x7f2c4c006210 comp rx=0 tx=0).stop 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c50106f70 msgr2=0x7f2c501a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c50106f70 0x7f2c501a2c10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800c7a0 tx=0x7f2c4800cab0 comp rx=0 tx=0).stop 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 shutdown_connections 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2c3c06dac0 0x7f2c3c06ff80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c5006a270 0x7f2c5019e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:37.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2c50106f70 0x7f2c501a2c10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:37.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.207+0000 7f2c579aa700 1 --2- 172.21.15.12:0/3697474675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c50104a80 0x7f2c5019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:37.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.208+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 >> 172.21.15.12:0/3697474675 conn(0x7f2c50077eb0 msgr2=0x7f2c500fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:37.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.208+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 shutdown_connections 2024-03-26T04:46:37.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:37.208+0000 7f2c579aa700 1 -- 172.21.15.12:0/3697474675 wait complete. 2024-03-26T04:46:37.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:37 smithi012 ceph-mon[29863]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:37.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1874328289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:46:37.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:37 smithi012 ceph-mon[41141]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:37.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1874328289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:46:37.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:37 smithi186 ceph-mon[33438]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:37.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1874328289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:46:37.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182441 for osd.3 2024-03-26T04:46:37.990 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:38.349 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3697474675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:46:38.349 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3697474675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:46:38.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3697474675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:46:39.072 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738424 for osd.0 2024-03-26T04:46:39.072 DEBUG:teuthology.parallel:result is None 2024-03-26T04:46:39.072 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T04:46:39.073 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:46:39.290 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:39 smithi012 ceph-mon[29863]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:39.290 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:39 smithi012 ceph-mon[41141]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:39.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:39 smithi186 ceph-mon[33438]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:39.567 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:40.655 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:40 smithi012 systemd[1]: Starting Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:46:41.452 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:41 smithi012 ceph-mon[29863]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:41.453 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:41 smithi012 ceph-mon[41141]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:41.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:41 smithi186 ceph-mon[33438]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 -- 172.21.15.12:0/1650948720 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda41080d0 msgr2=0x7efda41084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 --2- 172.21.15.12:0/1650948720 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda41080d0 0x7efda41084b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd940099c0 tx=0x7efd94009cd0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 -- 172.21.15.12:0/1650948720 shutdown_connections 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 --2- 172.21.15.12:0/1650948720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda406a6d0 0x7efda410ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 --2- 172.21.15.12:0/1650948720 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda41080d0 0x7efda41084b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 --2- 172.21.15.12:0/1650948720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda4069d30 0x7efda406a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 -- 172.21.15.12:0/1650948720 >> 172.21.15.12:0/1650948720 conn(0x7efda4077eb0 msgr2=0x7efda40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 -- 172.21.15.12:0/1650948720 shutdown_connections 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.650+0000 7efdaa350700 1 -- 172.21.15.12:0/1650948720 wait complete. 2024-03-26T04:46:41.651 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 Processor -- start 2024-03-26T04:46:41.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 -- start start 2024-03-26T04:46:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda4069d30 0x7efda419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:41.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 0x7efda419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 0x7efda4197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4197ec0 con 0x7efda41080d0 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4198000 con 0x7efda4069d30 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efdaa350700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda41981c0 con 0x7efda406a6d0 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda37fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 0x7efda419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda88ed700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 0x7efda4197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda37fe700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 0x7efda419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42868/0 (socket says 172.21.15.12:42868) 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda88ed700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 0x7efda4197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39474/0 (socket says 172.21.15.12:39474) 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda37fe700 1 -- 172.21.15.12:0/4229352579 learned_addr learned my addr 172.21.15.12:0/4229352579 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda3fff700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda4069d30 0x7efda419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.651+0000 7efda88ed700 1 -- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 msgr2=0x7efda419dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda88ed700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 0x7efda419dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda88ed700 1 -- 172.21.15.12:0/4229352579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda4069d30 msgr2=0x7efda419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda88ed700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda4069d30 0x7efda419d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda88ed700 1 -- 172.21.15.12:0/4229352579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c009c40 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda37fe700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 0x7efda419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda88ed700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 0x7efda4197860 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00efb0 tx=0x7efd9c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c003a00 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7efd9c00f040 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd940095f0 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c004430 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.652+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda41a4570 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.653+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda4068d40 con 0x7efda41080d0 2024-03-26T04:46:41.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.653+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7efd9c003b60 con 0x7efda41080d0 2024-03-26T04:46:41.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.654+0000 7efda17fa700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efd8c06dac0 0x7efd8c06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:41.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.654+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7efd9c08d4a0 con 0x7efda41080d0 2024-03-26T04:46:41.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.654+0000 7efda3fff700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efd8c06dac0 0x7efd8c06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:41.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.654+0000 7efda3fff700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efd8c06dac0 0x7efd8c06ff80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd94009fd0 tx=0x7efd94005fb0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:41.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.659+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efd9c056890 con 0x7efda41080d0 2024-03-26T04:46:41.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.862+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 --> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efda4111510 con 0x7efd8c06dac0 2024-03-26T04:46:41.874 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.873+0000 7efda17fa700 1 -- 172.21.15.12:0/4229352579 <== mgr.24464 v2:172.21.15.12:6800/715138906 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346508 (secure 0 0 0) 0x7efda4111510 con 0x7efd8c06dac0 2024-03-26T04:46:41.878 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:46:41.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efd8c06dac0 msgr2=0x7efd8c06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efd8c06dac0 0x7efd8c06ff80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd94009fd0 tx=0x7efd94005fb0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 msgr2=0x7efda4197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 0x7efda4197860 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00efb0 tx=0x7efd9c00c3a0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 shutdown_connections 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7efd8c06dac0 0x7efd8c06ff80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efda41080d0 0x7efda4197860 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efda4069d30 0x7efda419d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 --2- 172.21.15.12:0/4229352579 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7efda406a6d0 0x7efda419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 >> 172.21.15.12:0/4229352579 conn(0x7efda4077eb0 msgr2=0x7efda4105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:41.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 shutdown_connections 2024-03-26T04:46:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:41.876+0000 7efdaa350700 1 -- 172.21.15.12:0/4229352579 wait complete. 2024-03-26T04:46:41.883 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-26T04:46:42.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:41 smithi012 podman[76149]: 2024-03-26 04:46:41.746336616 +0000 UTC m=+0.018072955 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T04:46:42.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:41 smithi012 podman[76149]: 2024-03-26 04:46:41.860178696 +0000 UTC m=+0.131915020 volume create bdd23dc1eb1bd84593024465bf288bd45669a5d4d2d2351c6ef4d033439ac979 2024-03-26T04:46:42.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:41 smithi012 podman[76149]: 2024-03-26 04:46:41.967073687 +0000 UTC m=+0.238810053 container create 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:46:42.167 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:46:41 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:46:42.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-mon[29863]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:46:42.421 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:42.421 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-mon[41141]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:46:42.421 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:42.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:42 smithi186 ceph-mon[33438]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:46:42.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:42 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 podman[76149]: 2024-03-26 04:46:42.504152834 +0000 UTC m=+0.775889161 container init 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 podman[76149]: 2024-03-26 04:46:42.506758614 +0000 UTC m=+0.778494938 container start 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.518Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.519Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.520Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.12 port=9094 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.521Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:46:42.774 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.544Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:46:42.775 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.545Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:46:42.775 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.547Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:46:42.775 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:42.547Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:46:42.775 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 bash[76149]: 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 2024-03-26T04:46:42.775 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:42 smithi012 systemd[1]: Started Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:46:43.031 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:42] ENGINE Bus STOPPING 2024-03-26T04:46:43.031 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:46:43.031 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:42] ENGINE Bus STOPPED 2024-03-26T04:46:43.031 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:42] ENGINE Bus STARTING 2024-03-26T04:46:43.031 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:42] ENGINE Serving on http://:::9283 2024-03-26T04:46:43.032 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:42 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:46:42] ENGINE Bus STARTED 2024-03-26T04:46:43.033 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-26T04:46:40.094841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1291,"num_read_kb":1105,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1771764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236628,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724588,"statfs":{"total":767960285184,"available":767717978112,"internally_reserved":0,"allocated":8187904,"data_stored":4434810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188910+0000","last_change":"2024-03-26T04:45:26.786492+0000","last_active":"2024-03-26T04:46:00.188910+0000","last_peered":"2024-03-26T04:46:00.188910+0000","last_clean":"2024-03-26T04:46:00.188910+0000","last_became_active":"2024-03-26T04:45:26.786094+0000","last_became_peered":"2024-03-26T04:45:26.786094+0000","last_unstale":"2024-03-26T04:46:00.188910+0000","last_undegraded":"2024-03-26T04:46:00.188910+0000","last_fullsized":"2024-03-26T04:46:00.188910+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:49.907754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074989+0000","last_change":"2024-03-26T04:45:24.771946+0000","last_active":"2024-03-26T04:46:00.074989+0000","last_peered":"2024-03-26T04:46:00.074989+0000","last_clean":"2024-03-26T04:46:00.074989+0000","last_became_active":"2024-03-26T04:45:24.771633+0000","last_became_peered":"2024-03-26T04:45:24.771633+0000","last_unstale":"2024-03-26T04:46:00.074989+0000","last_undegraded":"2024-03-26T04:46:00.074989+0000","last_fullsized":"2024-03-26T04:46:00.074989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:06:27.751659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199569+0000","last_change":"2024-03-26T04:45:20.750121+0000","last_active":"2024-03-26T04:46:00.199569+0000","last_peered":"2024-03-26T04:46:00.199569+0000","last_clean":"2024-03-26T04:46:00.199569+0000","last_became_active":"2024-03-26T04:45:20.749745+0000","last_became_peered":"2024-03-26T04:45:20.749745+0000","last_unstale":"2024-03-26T04:46:00.199569+0000","last_undegraded":"2024-03-26T04:46:00.199569+0000","last_fullsized":"2024-03-26T04:46:00.199569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:30:39.908300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"67'39","reported_seq":102,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188678+0000","last_change":"2024-03-26T04:45:22.754431+0000","last_active":"2024-03-26T04:46:00.188678+0000","last_peered":"2024-03-26T04:46:00.188678+0000","last_clean":"2024-03-26T04:46:00.188678+0000","last_became_active":"2024-03-26T04:45:22.754121+0000","last_became_peered":"2024-03-26T04:45:22.754121+0000","last_unstale":"2024-03-26T04:46:00.188678+0000","last_undegraded":"2024-03-26T04:46:00.188678+0000","last_fullsized":"2024-03-26T04:46:00.188678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:18:36.681830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075201+0000","last_change":"2024-03-26T04:45:26.778088+0000","last_active":"2024-03-26T04:46:00.075201+0000","last_peered":"2024-03-26T04:46:00.075201+0000","last_clean":"2024-03-26T04:46:00.075201+0000","last_became_active":"2024-03-26T04:45:26.777655+0000","last_became_peered":"2024-03-26T04:45:26.777655+0000","last_unstale":"2024-03-26T04:46:00.075201+0000","last_undegraded":"2024-03-26T04:46:00.075201+0000","last_fullsized":"2024-03-26T04:46:00.075201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:03:57.686967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073263+0000","last_change":"2024-03-26T04:45:24.780540+0000","last_active":"2024-03-26T04:46:00.073263+0000","last_peered":"2024-03-26T04:46:00.073263+0000","last_clean":"2024-03-26T04:46:00.073263+0000","last_became_active":"2024-03-26T04:45:24.780166+0000","last_became_peered":"2024-03-26T04:45:24.780166+0000","last_unstale":"2024-03-26T04:46:00.073263+0000","last_undegraded":"2024-03-26T04:46:00.073263+0000","last_fullsized":"2024-03-26T04:46:00.073263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:15:33.027204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189809+0000","last_change":"2024-03-26T04:45:20.739973+0000","last_active":"2024-03-26T04:46:00.189809+0000","last_peered":"2024-03-26T04:46:00.189809+0000","last_clean":"2024-03-26T04:46:00.189809+0000","last_became_active":"2024-03-26T04:45:20.739684+0000","last_became_peered":"2024-03-26T04:45:20.739684+0000","last_unstale":"2024-03-26T04:46:00.189809+0000","last_undegraded":"2024-03-26T04:46:00.189809+0000","last_fullsized":"2024-03-26T04:46:00.189809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:51:56.871362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075133+0000","last_change":"2024-03-26T04:45:22.756717+0000","last_active":"2024-03-26T04:46:00.075133+0000","last_peered":"2024-03-26T04:46:00.075133+0000","last_clean":"2024-03-26T04:46:00.075133+0000","last_became_active":"2024-03-26T04:45:22.756401+0000","last_became_peered":"2024-03-26T04:45:22.756401+0000","last_unstale":"2024-03-26T04:46:00.075133+0000","last_undegraded":"2024-03-26T04:46:00.075133+0000","last_fullsized":"2024-03-26T04:46:00.075133+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:06:03.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469636+0000","last_change":"2024-03-26T04:45:26.785043+0000","last_active":"2024-03-26T04:46:00.469636+0000","last_peered":"2024-03-26T04:46:00.469636+0000","last_clean":"2024-03-26T04:46:00.469636+0000","last_became_active":"2024-03-26T04:45:26.784729+0000","last_became_peered":"2024-03-26T04:45:26.784729+0000","last_unstale":"2024-03-26T04:46:00.469636+0000","last_undegraded":"2024-03-26T04:46:00.469636+0000","last_fullsized":"2024-03-26T04:46:00.469636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:12:03.696267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470488+0000","last_change":"2024-03-26T04:45:24.779587+0000","last_active":"2024-03-26T04:46:00.470488+0000","last_peered":"2024-03-26T04:46:00.470488+0000","last_clean":"2024-03-26T04:46:00.470488+0000","last_became_active":"2024-03-26T04:45:24.779269+0000","last_became_peered":"2024-03-26T04:45:24.779269+0000","last_unstale":"2024-03-26T04:46:00.470488+0000","last_undegraded":"2024-03-26T04:46:00.470488+0000","last_fullsized":"2024-03-26T04:46:00.470488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:08:44.536805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471032+0000","last_change":"2024-03-26T04:45:20.750226+0000","last_active":"2024-03-26T04:46:00.471032+0000","last_peered":"2024-03-26T04:46:00.471032+0000","last_clean":"2024-03-26T04:46:00.471032+0000","last_became_active":"2024-03-26T04:45:20.749958+0000","last_became_peered":"2024-03-26T04:45:20.749958+0000","last_unstale":"2024-03-26T04:46:00.471032+0000","last_undegraded":"2024-03-26T04:46:00.471032+0000","last_fullsized":"2024-03-26T04:46:00.471032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:15:34.043794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075703+0000","last_change":"2024-03-26T04:45:22.757451+0000","last_active":"2024-03-26T04:46:00.075703+0000","last_peered":"2024-03-26T04:46:00.075703+0000","last_clean":"2024-03-26T04:46:00.075703+0000","last_became_active":"2024-03-26T04:45:22.757121+0000","last_became_peered":"2024-03-26T04:45:22.757121+0000","last_unstale":"2024-03-26T04:46:00.075703+0000","last_undegraded":"2024-03-26T04:46:00.075703+0000","last_fullsized":"2024-03-26T04:46:00.075703+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:43:14.048606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199206+0000","last_change":"2024-03-26T04:45:26.788470+0000","last_active":"2024-03-26T04:46:00.199206+0000","last_peered":"2024-03-26T04:46:00.199206+0000","last_clean":"2024-03-26T04:46:00.199206+0000","last_became_active":"2024-03-26T04:45:26.788138+0000","last_became_peered":"2024-03-26T04:45:26.788138+0000","last_unstale":"2024-03-26T04:46:00.199206+0000","last_undegraded":"2024-03-26T04:46:00.199206+0000","last_fullsized":"2024-03-26T04:46:00.199206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:24:44.757867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469833+0000","last_change":"2024-03-26T04:45:24.777381+0000","last_active":"2024-03-26T04:46:00.469833+0000","last_peered":"2024-03-26T04:46:00.469833+0000","last_clean":"2024-03-26T04:46:00.469833+0000","last_became_active":"2024-03-26T04:45:24.777084+0000","last_became_peered":"2024-03-26T04:45:24.777084+0000","last_unstale":"2024-03-26T04:46:00.469833+0000","last_undegraded":"2024-03-26T04:46:00.469833+0000","last_fullsized":"2024-03-26T04:46:00.469833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:54:36.974749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470644+0000","last_change":"2024-03-26T04:45:20.740965+0000","last_active":"2024-03-26T04:46:00.470644+0000","last_peered":"2024-03-26T04:46:00.470644+0000","last_clean":"2024-03-26T04:46:00.470644+0000","last_became_active":"2024-03-26T04:45:20.740678+0000","last_became_peered":"2024-03-26T04:45:20.740678+0000","last_unstale":"2024-03-26T04:46:00.470644+0000","last_undegraded":"2024-03-26T04:46:00.470644+0000","last_fullsized":"2024-03-26T04:46:00.470644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:09:57.474208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075302+0000","last_change":"2024-03-26T04:45:22.755814+0000","last_active":"2024-03-26T04:46:00.075302+0000","last_peered":"2024-03-26T04:46:00.075302+0000","last_clean":"2024-03-26T04:46:00.075302+0000","last_became_active":"2024-03-26T04:45:22.755475+0000","last_became_peered":"2024-03-26T04:45:22.755475+0000","last_unstale":"2024-03-26T04:46:00.075302+0000","last_undegraded":"2024-03-26T04:46:00.075302+0000","last_fullsized":"2024-03-26T04:46:00.075302+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:35:21.930949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189890+0000","last_change":"2024-03-26T04:45:26.788091+0000","last_active":"2024-03-26T04:46:00.189890+0000","last_peered":"2024-03-26T04:46:00.189890+0000","last_clean":"2024-03-26T04:46:00.189890+0000","last_became_active":"2024-03-26T04:45:26.787742+0000","last_became_peered":"2024-03-26T04:45:26.787742+0000","last_unstale":"2024-03-26T04:46:00.189890+0000","last_undegraded":"2024-03-26T04:46:00.189890+0000","last_fullsized":"2024-03-26T04:46:00.189890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:04:19.783833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075495+0000","last_change":"2024-03-26T04:45:24.775474+0000","last_active":"2024-03-26T04:46:00.075495+0000","last_peered":"2024-03-26T04:46:00.075495+0000","last_clean":"2024-03-26T04:46:00.075495+0000","last_became_active":"2024-03-26T04:45:24.775070+0000","last_became_peered":"2024-03-26T04:45:24.775070+0000","last_unstale":"2024-03-26T04:46:00.075495+0000","last_undegraded":"2024-03-26T04:46:00.075495+0000","last_fullsized":"2024-03-26T04:46:00.075495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:08:56.589055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189092+0000","last_change":"2024-03-26T04:45:20.738137+0000","last_active":"2024-03-26T04:46:00.189092+0000","last_peered":"2024-03-26T04:46:00.189092+0000","last_clean":"2024-03-26T04:46:00.189092+0000","last_became_active":"2024-03-26T04:45:20.737112+0000","last_became_peered":"2024-03-26T04:45:20.737112+0000","last_unstale":"2024-03-26T04:46:00.189092+0000","last_undegraded":"2024-03-26T04:46:00.189092+0000","last_fullsized":"2024-03-26T04:46:00.189092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:09:18.767305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":84,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189635+0000","last_change":"2024-03-26T04:45:45.921661+0000","last_active":"2024-03-26T04:46:00.189635+0000","last_peered":"2024-03-26T04:46:00.189635+0000","last_clean":"2024-03-26T04:46:00.189635+0000","last_became_active":"2024-03-26T04:45:45.906481+0000","last_became_peered":"2024-03-26T04:45:45.906481+0000","last_unstale":"2024-03-26T04:46:00.189635+0000","last_undegraded":"2024-03-26T04:46:00.189635+0000","last_fullsized":"2024-03-26T04:46:00.189635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:15:47.485396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076166+0000","last_change":"2024-03-26T04:45:26.782142+0000","last_active":"2024-03-26T04:46:00.076166+0000","last_peered":"2024-03-26T04:46:00.076166+0000","last_clean":"2024-03-26T04:46:00.076166+0000","last_became_active":"2024-03-26T04:45:26.781832+0000","last_became_peered":"2024-03-26T04:45:26.781832+0000","last_unstale":"2024-03-26T04:46:00.076166+0000","last_undegraded":"2024-03-26T04:46:00.076166+0000","last_fullsized":"2024-03-26T04:46:00.076166+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:52:01.699447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074168+0000","last_change":"2024-03-26T04:45:24.771980+0000","last_active":"2024-03-26T04:46:00.074168+0000","last_peered":"2024-03-26T04:46:00.074168+0000","last_clean":"2024-03-26T04:46:00.074168+0000","last_became_active":"2024-03-26T04:45:24.771665+0000","last_became_peered":"2024-03-26T04:45:24.771665+0000","last_unstale":"2024-03-26T04:46:00.074168+0000","last_undegraded":"2024-03-26T04:46:00.074168+0000","last_fullsized":"2024-03-26T04:46:00.074168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:39.810518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470809+0000","last_change":"2024-03-26T04:45:20.737703+0000","last_active":"2024-03-26T04:46:00.470809+0000","last_peered":"2024-03-26T04:46:00.470809+0000","last_clean":"2024-03-26T04:46:00.470809+0000","last_became_active":"2024-03-26T04:45:20.737397+0000","last_became_peered":"2024-03-26T04:45:20.737397+0000","last_unstale":"2024-03-26T04:46:00.470809+0000","last_undegraded":"2024-03-26T04:46:00.470809+0000","last_fullsized":"2024-03-26T04:46:00.470809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:39:52.818920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076803+0000","last_change":"2024-03-26T04:45:22.759227+0000","last_active":"2024-03-26T04:46:00.076803+0000","last_peered":"2024-03-26T04:46:00.076803+0000","last_clean":"2024-03-26T04:46:00.076803+0000","last_became_active":"2024-03-26T04:45:22.758820+0000","last_became_peered":"2024-03-26T04:45:22.758820+0000","last_unstale":"2024-03-26T04:46:00.076803+0000","last_undegraded":"2024-03-26T04:46:00.076803+0000","last_fullsized":"2024-03-26T04:46:00.076803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:22:42.965112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073752+0000","last_change":"2024-03-26T04:45:26.781913+0000","last_active":"2024-03-26T04:46:00.073752+0000","last_peered":"2024-03-26T04:46:00.073752+0000","last_clean":"2024-03-26T04:46:00.073752+0000","last_became_active":"2024-03-26T04:45:26.781562+0000","last_became_peered":"2024-03-26T04:45:26.781562+0000","last_unstale":"2024-03-26T04:46:00.073752+0000","last_undegraded":"2024-03-26T04:46:00.073752+0000","last_fullsized":"2024-03-26T04:46:00.073752+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:17:52.863809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.200011+0000","last_change":"2024-03-26T04:45:24.778698+0000","last_active":"2024-03-26T04:46:00.200011+0000","last_peered":"2024-03-26T04:46:00.200011+0000","last_clean":"2024-03-26T04:46:00.200011+0000","last_became_active":"2024-03-26T04:45:24.778388+0000","last_became_peered":"2024-03-26T04:45:24.778388+0000","last_unstale":"2024-03-26T04:46:00.200011+0000","last_undegraded":"2024-03-26T04:46:00.200011+0000","last_fullsized":"2024-03-26T04:46:00.200011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:19:09.788321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188373+0000","last_change":"2024-03-26T04:45:20.739547+0000","last_active":"2024-03-26T04:46:00.188373+0000","last_peered":"2024-03-26T04:46:00.188373+0000","last_clean":"2024-03-26T04:46:00.188373+0000","last_became_active":"2024-03-26T04:45:20.739256+0000","last_became_peered":"2024-03-26T04:45:20.739256+0000","last_unstale":"2024-03-26T04:46:00.188373+0000","last_undegraded":"2024-03-26T04:46:00.188373+0000","last_fullsized":"2024-03-26T04:46:00.188373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:34:23.837482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470708+0000","last_change":"2024-03-26T04:45:22.761083+0000","last_active":"2024-03-26T04:46:00.470708+0000","last_peered":"2024-03-26T04:46:00.470708+0000","last_clean":"2024-03-26T04:46:00.470708+0000","last_became_active":"2024-03-26T04:45:22.760776+0000","last_became_peered":"2024-03-26T04:45:22.760776+0000","last_unstale":"2024-03-26T04:46:00.470708+0000","last_undegraded":"2024-03-26T04:46:00.470708+0000","last_fullsized":"2024-03-26T04:46:00.470708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:59:52.847474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470469+0000","last_change":"2024-03-26T04:45:26.788091+0000","last_active":"2024-03-26T04:46:00.470469+0000","last_peered":"2024-03-26T04:46:00.470469+0000","last_clean":"2024-03-26T04:46:00.470469+0000","last_became_active":"2024-03-26T04:45:26.787780+0000","last_became_peered":"2024-03-26T04:45:26.787780+0000","last_unstale":"2024-03-26T04:46:00.470469+0000","last_undegraded":"2024-03-26T04:46:00.470469+0000","last_fullsized":"2024-03-26T04:46:00.470469+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:26:45.315783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805874+0000","last_change":"2024-03-26T04:45:24.779611+0000","last_active":"2024-03-26T04:46:30.805874+0000","last_peered":"2024-03-26T04:46:30.805874+0000","last_clean":"2024-03-26T04:46:30.805874+0000","last_became_active":"2024-03-26T04:45:24.778211+0000","last_became_peered":"2024-03-26T04:45:24.778211+0000","last_unstale":"2024-03-26T04:46:30.805874+0000","last_undegraded":"2024-03-26T04:46:30.805874+0000","last_fullsized":"2024-03-26T04:46:30.805874+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:01:20.063792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469283+0000","last_change":"2024-03-26T04:45:20.741997+0000","last_active":"2024-03-26T04:46:00.469283+0000","last_peered":"2024-03-26T04:46:00.469283+0000","last_clean":"2024-03-26T04:46:00.469283+0000","last_became_active":"2024-03-26T04:45:20.741681+0000","last_became_peered":"2024-03-26T04:45:20.741681+0000","last_unstale":"2024-03-26T04:46:00.469283+0000","last_undegraded":"2024-03-26T04:46:00.469283+0000","last_fullsized":"2024-03-26T04:46:00.469283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:34:33.166949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199995+0000","last_change":"2024-03-26T04:45:22.750286+0000","last_active":"2024-03-26T04:46:00.199995+0000","last_peered":"2024-03-26T04:46:00.199995+0000","last_clean":"2024-03-26T04:46:00.199995+0000","last_became_active":"2024-03-26T04:45:22.749925+0000","last_became_peered":"2024-03-26T04:45:22.749925+0000","last_unstale":"2024-03-26T04:46:00.199995+0000","last_undegraded":"2024-03-26T04:46:00.199995+0000","last_fullsized":"2024-03-26T04:46:00.199995+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:06:20.335133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189228+0000","last_change":"2024-03-26T04:45:26.783513+0000","last_active":"2024-03-26T04:46:00.189228+0000","last_peered":"2024-03-26T04:46:00.189228+0000","last_clean":"2024-03-26T04:46:00.189228+0000","last_became_active":"2024-03-26T04:45:26.783159+0000","last_became_peered":"2024-03-26T04:45:26.783159+0000","last_unstale":"2024-03-26T04:46:00.189228+0000","last_undegraded":"2024-03-26T04:46:00.189228+0000","last_fullsized":"2024-03-26T04:46:00.189228+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:55.621387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471031+0000","last_change":"2024-03-26T04:45:24.781221+0000","last_active":"2024-03-26T04:46:00.471031+0000","last_peered":"2024-03-26T04:46:00.471031+0000","last_clean":"2024-03-26T04:46:00.471031+0000","last_became_active":"2024-03-26T04:45:24.779992+0000","last_became_peered":"2024-03-26T04:45:24.779992+0000","last_unstale":"2024-03-26T04:46:00.471031+0000","last_undegraded":"2024-03-26T04:46:00.471031+0000","last_fullsized":"2024-03-26T04:46:00.471031+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:53:59.760550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471048+0000","last_change":"2024-03-26T04:45:20.739468+0000","last_active":"2024-03-26T04:46:00.471048+0000","last_peered":"2024-03-26T04:46:00.471048+0000","last_clean":"2024-03-26T04:46:00.471048+0000","last_became_active":"2024-03-26T04:45:20.739162+0000","last_became_peered":"2024-03-26T04:45:20.739162+0000","last_unstale":"2024-03-26T04:46:00.471048+0000","last_undegraded":"2024-03-26T04:46:00.471048+0000","last_fullsized":"2024-03-26T04:46:00.471048+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:20.871210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188658+0000","last_change":"2024-03-26T04:45:22.758910+0000","last_active":"2024-03-26T04:46:00.188658+0000","last_peered":"2024-03-26T04:46:00.188658+0000","last_clean":"2024-03-26T04:46:00.188658+0000","last_became_active":"2024-03-26T04:45:22.758588+0000","last_became_peered":"2024-03-26T04:45:22.758588+0000","last_unstale":"2024-03-26T04:46:00.188658+0000","last_undegraded":"2024-03-26T04:46:00.188658+0000","last_fullsized":"2024-03-26T04:46:00.188658+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:28:00.884941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470309+0000","last_change":"2024-03-26T04:45:26.785967+0000","last_active":"2024-03-26T04:46:00.470309+0000","last_peered":"2024-03-26T04:46:00.470309+0000","last_clean":"2024-03-26T04:46:00.470309+0000","last_became_active":"2024-03-26T04:45:26.785682+0000","last_became_peered":"2024-03-26T04:45:26.785682+0000","last_unstale":"2024-03-26T04:46:00.470309+0000","last_undegraded":"2024-03-26T04:46:00.470309+0000","last_fullsized":"2024-03-26T04:46:00.470309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:09:32.286415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469540+0000","last_change":"2024-03-26T04:45:24.774208+0000","last_active":"2024-03-26T04:46:00.469540+0000","last_peered":"2024-03-26T04:46:00.469540+0000","last_clean":"2024-03-26T04:46:00.469540+0000","last_became_active":"2024-03-26T04:45:24.773883+0000","last_became_peered":"2024-03-26T04:45:24.773883+0000","last_unstale":"2024-03-26T04:46:00.469540+0000","last_undegraded":"2024-03-26T04:46:00.469540+0000","last_fullsized":"2024-03-26T04:46:00.469540+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:45.352012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471123+0000","last_change":"2024-03-26T04:45:20.741746+0000","last_active":"2024-03-26T04:46:00.471123+0000","last_peered":"2024-03-26T04:46:00.471123+0000","last_clean":"2024-03-26T04:46:00.471123+0000","last_became_active":"2024-03-26T04:45:20.741165+0000","last_became_peered":"2024-03-26T04:45:20.741165+0000","last_unstale":"2024-03-26T04:46:00.471123+0000","last_undegraded":"2024-03-26T04:46:00.471123+0000","last_fullsized":"2024-03-26T04:46:00.471123+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:19:04.836421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188288+0000","last_change":"2024-03-26T04:45:22.755755+0000","last_active":"2024-03-26T04:46:00.188288+0000","last_peered":"2024-03-26T04:46:00.188288+0000","last_clean":"2024-03-26T04:46:00.188288+0000","last_became_active":"2024-03-26T04:45:22.755440+0000","last_became_peered":"2024-03-26T04:45:22.755440+0000","last_unstale":"2024-03-26T04:46:00.188288+0000","last_undegraded":"2024-03-26T04:46:00.188288+0000","last_fullsized":"2024-03-26T04:46:00.188288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:58.905337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188403+0000","last_change":"2024-03-26T04:45:26.788244+0000","last_active":"2024-03-26T04:46:00.188403+0000","last_peered":"2024-03-26T04:46:00.188403+0000","last_clean":"2024-03-26T04:46:00.188403+0000","last_became_active":"2024-03-26T04:45:26.787918+0000","last_became_peered":"2024-03-26T04:45:26.787918+0000","last_unstale":"2024-03-26T04:46:00.188403+0000","last_undegraded":"2024-03-26T04:46:00.188403+0000","last_fullsized":"2024-03-26T04:46:00.188403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:56:19.821640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075303+0000","last_change":"2024-03-26T04:45:24.780301+0000","last_active":"2024-03-26T04:46:00.075303+0000","last_peered":"2024-03-26T04:46:00.075303+0000","last_clean":"2024-03-26T04:46:00.075303+0000","last_became_active":"2024-03-26T04:45:24.779817+0000","last_became_peered":"2024-03-26T04:45:24.779817+0000","last_unstale":"2024-03-26T04:46:00.075303+0000","last_undegraded":"2024-03-26T04:46:00.075303+0000","last_fullsized":"2024-03-26T04:46:00.075303+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:28:15.892797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073977+0000","last_change":"2024-03-26T04:45:20.749055+0000","last_active":"2024-03-26T04:46:00.073977+0000","last_peered":"2024-03-26T04:46:00.073977+0000","last_clean":"2024-03-26T04:46:00.073977+0000","last_became_active":"2024-03-26T04:45:20.748715+0000","last_became_peered":"2024-03-26T04:45:20.748715+0000","last_unstale":"2024-03-26T04:46:00.073977+0000","last_undegraded":"2024-03-26T04:46:00.073977+0000","last_fullsized":"2024-03-26T04:46:00.073977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:54.073215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075321+0000","last_change":"2024-03-26T04:45:22.760671+0000","last_active":"2024-03-26T04:46:00.075321+0000","last_peered":"2024-03-26T04:46:00.075321+0000","last_clean":"2024-03-26T04:46:00.075321+0000","last_became_active":"2024-03-26T04:45:22.760306+0000","last_became_peered":"2024-03-26T04:45:22.760306+0000","last_unstale":"2024-03-26T04:46:00.075321+0000","last_undegraded":"2024-03-26T04:46:00.075321+0000","last_fullsized":"2024-03-26T04:46:00.075321+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:23:10.513174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.198789+0000","last_change":"2024-03-26T04:45:26.790448+0000","last_active":"2024-03-26T04:46:00.198789+0000","last_peered":"2024-03-26T04:46:00.198789+0000","last_clean":"2024-03-26T04:46:00.198789+0000","last_became_active":"2024-03-26T04:45:26.790096+0000","last_became_peered":"2024-03-26T04:45:26.790096+0000","last_unstale":"2024-03-26T04:46:00.198789+0000","last_undegraded":"2024-03-26T04:46:00.198789+0000","last_fullsized":"2024-03-26T04:46:00.198789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:31.119087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189142+0000","last_change":"2024-03-26T04:45:24.780791+0000","last_active":"2024-03-26T04:46:00.189142+0000","last_peered":"2024-03-26T04:46:00.189142+0000","last_clean":"2024-03-26T04:46:00.189142+0000","last_became_active":"2024-03-26T04:45:24.780466+0000","last_became_peered":"2024-03-26T04:45:24.780466+0000","last_unstale":"2024-03-26T04:46:00.189142+0000","last_undegraded":"2024-03-26T04:46:00.189142+0000","last_fullsized":"2024-03-26T04:46:00.189142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:42:04.128173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471244+0000","last_change":"2024-03-26T04:45:20.740613+0000","last_active":"2024-03-26T04:46:00.471244+0000","last_peered":"2024-03-26T04:46:00.471244+0000","last_clean":"2024-03-26T04:46:00.471244+0000","last_became_active":"2024-03-26T04:45:20.740304+0000","last_became_peered":"2024-03-26T04:45:20.740304+0000","last_unstale":"2024-03-26T04:46:00.471244+0000","last_undegraded":"2024-03-26T04:46:00.471244+0000","last_fullsized":"2024-03-26T04:46:00.471244+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:05:04.102035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075139+0000","last_change":"2024-03-26T04:45:22.757790+0000","last_active":"2024-03-26T04:46:00.075139+0000","last_peered":"2024-03-26T04:46:00.075139+0000","last_clean":"2024-03-26T04:46:00.075139+0000","last_became_active":"2024-03-26T04:45:22.756742+0000","last_became_peered":"2024-03-26T04:45:22.756742+0000","last_unstale":"2024-03-26T04:46:00.075139+0000","last_undegraded":"2024-03-26T04:46:00.075139+0000","last_fullsized":"2024-03-26T04:46:00.075139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:12:27.104263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076003+0000","last_change":"2024-03-26T04:45:26.777995+0000","last_active":"2024-03-26T04:46:00.076003+0000","last_peered":"2024-03-26T04:46:00.076003+0000","last_clean":"2024-03-26T04:46:00.076003+0000","last_became_active":"2024-03-26T04:45:26.777394+0000","last_became_peered":"2024-03-26T04:45:26.777394+0000","last_unstale":"2024-03-26T04:46:00.076003+0000","last_undegraded":"2024-03-26T04:46:00.076003+0000","last_fullsized":"2024-03-26T04:46:00.076003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:19:53.611019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805507+0000","last_change":"2024-03-26T04:45:24.781638+0000","last_active":"2024-03-26T04:46:30.805507+0000","last_peered":"2024-03-26T04:46:30.805507+0000","last_clean":"2024-03-26T04:46:30.805507+0000","last_became_active":"2024-03-26T04:45:24.781269+0000","last_became_peered":"2024-03-26T04:45:24.781269+0000","last_unstale":"2024-03-26T04:46:30.805507+0000","last_undegraded":"2024-03-26T04:46:30.805507+0000","last_fullsized":"2024-03-26T04:46:30.805507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:59:41.625581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199497+0000","last_change":"2024-03-26T04:45:20.744611+0000","last_active":"2024-03-26T04:46:00.199497+0000","last_peered":"2024-03-26T04:46:00.199497+0000","last_clean":"2024-03-26T04:46:00.199497+0000","last_became_active":"2024-03-26T04:45:20.744113+0000","last_became_peered":"2024-03-26T04:45:20.744113+0000","last_unstale":"2024-03-26T04:46:00.199497+0000","last_undegraded":"2024-03-26T04:46:00.199497+0000","last_fullsized":"2024-03-26T04:46:00.199497+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:59.753246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471352+0000","last_change":"2024-03-26T04:45:22.753202+0000","last_active":"2024-03-26T04:46:00.471352+0000","last_peered":"2024-03-26T04:46:00.471352+0000","last_clean":"2024-03-26T04:46:00.471352+0000","last_became_active":"2024-03-26T04:45:22.752239+0000","last_became_peered":"2024-03-26T04:45:22.752239+0000","last_unstale":"2024-03-26T04:46:00.471352+0000","last_undegraded":"2024-03-26T04:46:00.471352+0000","last_fullsized":"2024-03-26T04:46:00.471352+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:40:52.537201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199732+0000","last_change":"2024-03-26T04:45:26.789517+0000","last_active":"2024-03-26T04:46:00.199732+0000","last_peered":"2024-03-26T04:46:00.199732+0000","last_clean":"2024-03-26T04:46:00.199732+0000","last_became_active":"2024-03-26T04:45:26.789195+0000","last_became_peered":"2024-03-26T04:45:26.789195+0000","last_unstale":"2024-03-26T04:46:00.199732+0000","last_undegraded":"2024-03-26T04:46:00.199732+0000","last_fullsized":"2024-03-26T04:46:00.199732+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:11:12.114878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805797+0000","last_change":"2024-03-26T04:45:24.784181+0000","last_active":"2024-03-26T04:46:30.805797+0000","last_peered":"2024-03-26T04:46:30.805797+0000","last_clean":"2024-03-26T04:46:30.805797+0000","last_became_active":"2024-03-26T04:45:24.783869+0000","last_became_peered":"2024-03-26T04:45:24.783869+0000","last_unstale":"2024-03-26T04:46:30.805797+0000","last_undegraded":"2024-03-26T04:46:30.805797+0000","last_fullsized":"2024-03-26T04:46:30.805797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:43:28.834081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469850+0000","last_change":"2024-03-26T04:45:20.740730+0000","last_active":"2024-03-26T04:46:00.469850+0000","last_peered":"2024-03-26T04:46:00.469850+0000","last_clean":"2024-03-26T04:46:00.469850+0000","last_became_active":"2024-03-26T04:45:20.739873+0000","last_became_peered":"2024-03-26T04:45:20.739873+0000","last_unstale":"2024-03-26T04:46:00.469850+0000","last_undegraded":"2024-03-26T04:46:00.469850+0000","last_fullsized":"2024-03-26T04:46:00.469850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:48:55.926794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189400+0000","last_change":"2024-03-26T04:45:22.761397+0000","last_active":"2024-03-26T04:46:00.189400+0000","last_peered":"2024-03-26T04:46:00.189400+0000","last_clean":"2024-03-26T04:46:00.189400+0000","last_became_active":"2024-03-26T04:45:22.760937+0000","last_became_peered":"2024-03-26T04:45:22.760937+0000","last_unstale":"2024-03-26T04:46:00.189400+0000","last_undegraded":"2024-03-26T04:46:00.189400+0000","last_fullsized":"2024-03-26T04:46:00.189400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:40:01.328517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469911+0000","last_change":"2024-03-26T04:45:26.782497+0000","last_active":"2024-03-26T04:46:00.469911+0000","last_peered":"2024-03-26T04:46:00.469911+0000","last_clean":"2024-03-26T04:46:00.469911+0000","last_became_active":"2024-03-26T04:45:26.782215+0000","last_became_peered":"2024-03-26T04:45:26.782215+0000","last_unstale":"2024-03-26T04:46:00.469911+0000","last_undegraded":"2024-03-26T04:46:00.469911+0000","last_fullsized":"2024-03-26T04:46:00.469911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:13.088620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469833+0000","last_change":"2024-03-26T04:45:24.784123+0000","last_active":"2024-03-26T04:46:00.469833+0000","last_peered":"2024-03-26T04:46:00.469833+0000","last_clean":"2024-03-26T04:46:00.469833+0000","last_became_active":"2024-03-26T04:45:24.783786+0000","last_became_peered":"2024-03-26T04:45:24.783786+0000","last_unstale":"2024-03-26T04:46:00.469833+0000","last_undegraded":"2024-03-26T04:46:00.469833+0000","last_fullsized":"2024-03-26T04:46:00.469833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:52:14.978506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469648+0000","last_change":"2024-03-26T04:45:20.736887+0000","last_active":"2024-03-26T04:46:00.469648+0000","last_peered":"2024-03-26T04:46:00.469648+0000","last_clean":"2024-03-26T04:46:00.469648+0000","last_became_active":"2024-03-26T04:45:20.736259+0000","last_became_peered":"2024-03-26T04:45:20.736259+0000","last_unstale":"2024-03-26T04:46:00.469648+0000","last_undegraded":"2024-03-26T04:46:00.469648+0000","last_fullsized":"2024-03-26T04:46:00.469648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:10:39.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189850+0000","last_change":"2024-03-26T04:45:22.757325+0000","last_active":"2024-03-26T04:46:00.189850+0000","last_peered":"2024-03-26T04:46:00.189850+0000","last_clean":"2024-03-26T04:46:00.189850+0000","last_became_active":"2024-03-26T04:45:22.756989+0000","last_became_peered":"2024-03-26T04:45:22.756989+0000","last_unstale":"2024-03-26T04:46:00.189850+0000","last_undegraded":"2024-03-26T04:46:00.189850+0000","last_fullsized":"2024-03-26T04:46:00.189850+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:48:32.854181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076259+0000","last_change":"2024-03-26T04:45:26.780364+0000","last_active":"2024-03-26T04:46:00.076259+0000","last_peered":"2024-03-26T04:46:00.076259+0000","last_clean":"2024-03-26T04:46:00.076259+0000","last_became_active":"2024-03-26T04:45:26.780016+0000","last_became_peered":"2024-03-26T04:45:26.780016+0000","last_unstale":"2024-03-26T04:46:00.076259+0000","last_undegraded":"2024-03-26T04:46:00.076259+0000","last_fullsized":"2024-03-26T04:46:00.076259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:04:24.907474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189970+0000","last_change":"2024-03-26T04:45:24.775319+0000","last_active":"2024-03-26T04:46:00.189970+0000","last_peered":"2024-03-26T04:46:00.189970+0000","last_clean":"2024-03-26T04:46:00.189970+0000","last_became_active":"2024-03-26T04:45:24.775001+0000","last_became_peered":"2024-03-26T04:45:24.775001+0000","last_unstale":"2024-03-26T04:46:00.189970+0000","last_undegraded":"2024-03-26T04:46:00.189970+0000","last_fullsized":"2024-03-26T04:46:00.189970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:16:27.717843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075909+0000","last_change":"2024-03-26T04:45:20.746067+0000","last_active":"2024-03-26T04:46:00.075909+0000","last_peered":"2024-03-26T04:46:00.075909+0000","last_clean":"2024-03-26T04:46:00.075909+0000","last_became_active":"2024-03-26T04:45:20.745745+0000","last_became_peered":"2024-03-26T04:45:20.745745+0000","last_unstale":"2024-03-26T04:46:00.075909+0000","last_undegraded":"2024-03-26T04:46:00.075909+0000","last_fullsized":"2024-03-26T04:46:00.075909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:51:49.254140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199859+0000","last_change":"2024-03-26T04:45:22.750074+0000","last_active":"2024-03-26T04:46:00.199859+0000","last_peered":"2024-03-26T04:46:00.199859+0000","last_clean":"2024-03-26T04:46:00.199859+0000","last_became_active":"2024-03-26T04:45:22.749633+0000","last_became_peered":"2024-03-26T04:45:22.749633+0000","last_unstale":"2024-03-26T04:46:00.199859+0000","last_undegraded":"2024-03-26T04:46:00.199859+0000","last_fullsized":"2024-03-26T04:46:00.199859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:30:18.952438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189188+0000","last_change":"2024-03-26T04:45:26.783345+0000","last_active":"2024-03-26T04:46:00.189188+0000","last_peered":"2024-03-26T04:46:00.189188+0000","last_clean":"2024-03-26T04:46:00.189188+0000","last_became_active":"2024-03-26T04:45:26.782972+0000","last_became_peered":"2024-03-26T04:45:26.782972+0000","last_unstale":"2024-03-26T04:46:00.189188+0000","last_undegraded":"2024-03-26T04:46:00.189188+0000","last_fullsized":"2024-03-26T04:46:00.189188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:21:49.919377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076692+0000","last_change":"2024-03-26T04:45:20.744069+0000","last_active":"2024-03-26T04:46:00.076692+0000","last_peered":"2024-03-26T04:46:00.076692+0000","last_clean":"2024-03-26T04:46:00.076692+0000","last_became_active":"2024-03-26T04:45:20.743767+0000","last_became_peered":"2024-03-26T04:45:20.743767+0000","last_unstale":"2024-03-26T04:46:00.076692+0000","last_undegraded":"2024-03-26T04:46:00.076692+0000","last_fullsized":"2024-03-26T04:46:00.076692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:15.121354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075184+0000","last_change":"2024-03-26T04:45:22.756536+0000","last_active":"2024-03-26T04:46:00.075184+0000","last_peered":"2024-03-26T04:46:00.075184+0000","last_clean":"2024-03-26T04:46:00.075184+0000","last_became_active":"2024-03-26T04:45:22.756239+0000","last_became_peered":"2024-03-26T04:45:22.756239+0000","last_unstale":"2024-03-26T04:46:00.075184+0000","last_undegraded":"2024-03-26T04:46:00.075184+0000","last_fullsized":"2024-03-26T04:46:00.075184+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:27:46.056197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075820+0000","last_change":"2024-03-26T04:45:24.772349+0000","last_active":"2024-03-26T04:46:00.075820+0000","last_peered":"2024-03-26T04:46:00.075820+0000","last_clean":"2024-03-26T04:46:00.075820+0000","last_became_active":"2024-03-26T04:45:24.771977+0000","last_became_peered":"2024-03-26T04:45:24.771977+0000","last_unstale":"2024-03-26T04:46:00.075820+0000","last_undegraded":"2024-03-26T04:46:00.075820+0000","last_fullsized":"2024-03-26T04:46:00.075820+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:22:53.791394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.198801+0000","last_change":"2024-03-26T04:45:26.784601+0000","last_active":"2024-03-26T04:46:00.198801+0000","last_peered":"2024-03-26T04:46:00.198801+0000","last_clean":"2024-03-26T04:46:00.198801+0000","last_became_active":"2024-03-26T04:45:26.784262+0000","last_became_peered":"2024-03-26T04:45:26.784262+0000","last_unstale":"2024-03-26T04:46:00.198801+0000","last_undegraded":"2024-03-26T04:46:00.198801+0000","last_fullsized":"2024-03-26T04:46:00.198801+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:32.073700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073918+0000","last_change":"2024-03-26T04:45:20.748989+0000","last_active":"2024-03-26T04:46:00.073918+0000","last_peered":"2024-03-26T04:46:00.073918+0000","last_clean":"2024-03-26T04:46:00.073918+0000","last_became_active":"2024-03-26T04:45:20.748624+0000","last_became_peered":"2024-03-26T04:45:20.748624+0000","last_unstale":"2024-03-26T04:46:00.073918+0000","last_undegraded":"2024-03-26T04:46:00.073918+0000","last_fullsized":"2024-03-26T04:46:00.073918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:23:13.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075267+0000","last_change":"2024-03-26T04:45:22.756873+0000","last_active":"2024-03-26T04:46:00.075267+0000","last_peered":"2024-03-26T04:46:00.075267+0000","last_clean":"2024-03-26T04:46:00.075267+0000","last_became_active":"2024-03-26T04:45:22.756184+0000","last_became_peered":"2024-03-26T04:45:22.756184+0000","last_unstale":"2024-03-26T04:46:00.075267+0000","last_undegraded":"2024-03-26T04:46:00.075267+0000","last_fullsized":"2024-03-26T04:46:00.075267+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:22:38.661330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805491+0000","last_change":"2024-03-26T04:45:24.776787+0000","last_active":"2024-03-26T04:46:30.805491+0000","last_peered":"2024-03-26T04:46:30.805491+0000","last_clean":"2024-03-26T04:46:30.805491+0000","last_became_active":"2024-03-26T04:45:24.776172+0000","last_became_peered":"2024-03-26T04:45:24.776172+0000","last_unstale":"2024-03-26T04:46:30.805491+0000","last_undegraded":"2024-03-26T04:46:30.805491+0000","last_fullsized":"2024-03-26T04:46:30.805491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:40:30.435694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189810+0000","last_change":"2024-03-26T04:45:26.781682+0000","last_active":"2024-03-26T04:46:00.189810+0000","last_peered":"2024-03-26T04:46:00.189810+0000","last_clean":"2024-03-26T04:46:00.189810+0000","last_became_active":"2024-03-26T04:45:26.781280+0000","last_became_peered":"2024-03-26T04:45:26.781280+0000","last_unstale":"2024-03-26T04:46:00.189810+0000","last_undegraded":"2024-03-26T04:46:00.189810+0000","last_fullsized":"2024-03-26T04:46:00.189810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:20:27.101384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469946+0000","last_change":"2024-03-26T04:45:20.742917+0000","last_active":"2024-03-26T04:46:00.469946+0000","last_peered":"2024-03-26T04:46:00.469946+0000","last_clean":"2024-03-26T04:46:00.469946+0000","last_became_active":"2024-03-26T04:45:20.742643+0000","last_became_peered":"2024-03-26T04:45:20.742643+0000","last_unstale":"2024-03-26T04:46:00.469946+0000","last_undegraded":"2024-03-26T04:46:00.469946+0000","last_fullsized":"2024-03-26T04:46:00.469946+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:33:55.822942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'34","reported_seq":105,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805505+0000","last_change":"2024-03-26T04:45:22.761391+0000","last_active":"2024-03-26T04:46:30.805505+0000","last_peered":"2024-03-26T04:46:30.805505+0000","last_clean":"2024-03-26T04:46:30.805505+0000","last_became_active":"2024-03-26T04:45:22.761021+0000","last_became_peered":"2024-03-26T04:45:22.761021+0000","last_unstale":"2024-03-26T04:46:30.805505+0000","last_undegraded":"2024-03-26T04:46:30.805505+0000","last_fullsized":"2024-03-26T04:46:30.805505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:14:47.623733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199993+0000","last_change":"2024-03-26T04:45:24.777638+0000","last_active":"2024-03-26T04:46:00.199993+0000","last_peered":"2024-03-26T04:46:00.199993+0000","last_clean":"2024-03-26T04:46:00.199993+0000","last_became_active":"2024-03-26T04:45:24.777326+0000","last_became_peered":"2024-03-26T04:45:24.777326+0000","last_unstale":"2024-03-26T04:46:00.199993+0000","last_undegraded":"2024-03-26T04:46:00.199993+0000","last_fullsized":"2024-03-26T04:46:00.199993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:05:57.136042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471480+0000","last_change":"2024-03-26T04:45:26.785143+0000","last_active":"2024-03-26T04:46:00.471480+0000","last_peered":"2024-03-26T04:46:00.471480+0000","last_clean":"2024-03-26T04:46:00.471480+0000","last_became_active":"2024-03-26T04:45:26.784843+0000","last_became_peered":"2024-03-26T04:45:26.784843+0000","last_unstale":"2024-03-26T04:46:00.471480+0000","last_undegraded":"2024-03-26T04:46:00.471480+0000","last_fullsized":"2024-03-26T04:46:00.471480+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:37:00.953324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'5","reported_seq":113,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:34.925234+0000","last_change":"2024-03-26T04:45:24.765271+0000","last_active":"2024-03-26T04:46:34.925234+0000","last_peered":"2024-03-26T04:46:34.925234+0000","last_clean":"2024-03-26T04:46:34.925234+0000","last_became_active":"2024-03-26T04:45:20.735553+0000","last_became_peered":"2024-03-26T04:45:20.735553+0000","last_unstale":"2024-03-26T04:46:34.925234+0000","last_undegraded":"2024-03-26T04:46:34.925234+0000","last_fullsized":"2024-03-26T04:46:34.925234+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:42.388391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030280620000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":65,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075979+0000","last_change":"2024-03-26T04:45:20.740834+0000","last_active":"2024-03-26T04:46:00.075979+0000","last_peered":"2024-03-26T04:46:00.075979+0000","last_clean":"2024-03-26T04:46:00.075979+0000","last_became_active":"2024-03-26T04:45:20.740528+0000","last_became_peered":"2024-03-26T04:45:20.740528+0000","last_unstale":"2024-03-26T04:46:00.075979+0000","last_undegraded":"2024-03-26T04:46:00.075979+0000","last_fullsized":"2024-03-26T04:46:00.075979+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:35:07.801959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":27,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076626+0000","last_change":"2024-03-26T04:45:45.921771+0000","last_active":"2024-03-26T04:46:00.076626+0000","last_peered":"2024-03-26T04:46:00.076626+0000","last_clean":"2024-03-26T04:46:00.076626+0000","last_became_active":"2024-03-26T04:45:45.906012+0000","last_became_peered":"2024-03-26T04:45:45.906012+0000","last_unstale":"2024-03-26T04:46:00.076626+0000","last_undegraded":"2024-03-26T04:46:00.076626+0000","last_fullsized":"2024-03-26T04:46:00.076626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:14:59.192089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076569+0000","last_change":"2024-03-26T04:45:24.779099+0000","last_active":"2024-03-26T04:46:00.076569+0000","last_peered":"2024-03-26T04:46:00.076569+0000","last_clean":"2024-03-26T04:46:00.076569+0000","last_became_active":"2024-03-26T04:45:24.778722+0000","last_became_peered":"2024-03-26T04:45:24.778722+0000","last_unstale":"2024-03-26T04:46:00.076569+0000","last_undegraded":"2024-03-26T04:46:00.076569+0000","last_fullsized":"2024-03-26T04:46:00.076569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:47.337053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470905+0000","last_change":"2024-03-26T04:45:26.786252+0000","last_active":"2024-03-26T04:46:00.470905+0000","last_peered":"2024-03-26T04:46:00.470905+0000","last_clean":"2024-03-26T04:46:00.470905+0000","last_became_active":"2024-03-26T04:45:26.785929+0000","last_became_peered":"2024-03-26T04:45:26.785929+0000","last_unstale":"2024-03-26T04:46:00.470905+0000","last_undegraded":"2024-03-26T04:46:00.470905+0000","last_fullsized":"2024-03-26T04:46:00.470905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:36:23.721970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188410+0000","last_change":"2024-03-26T04:45:24.771645+0000","last_active":"2024-03-26T04:46:00.188410+0000","last_peered":"2024-03-26T04:46:00.188410+0000","last_clean":"2024-03-26T04:46:00.188410+0000","last_became_active":"2024-03-26T04:45:20.738369+0000","last_became_peered":"2024-03-26T04:45:20.738369+0000","last_unstale":"2024-03-26T04:46:00.188410+0000","last_undegraded":"2024-03-26T04:46:00.188410+0000","last_fullsized":"2024-03-26T04:46:00.188410+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:19:57.782576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020341589999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469742+0000","last_change":"2024-03-26T04:45:20.743758+0000","last_active":"2024-03-26T04:46:00.469742+0000","last_peered":"2024-03-26T04:46:00.469742+0000","last_clean":"2024-03-26T04:46:00.469742+0000","last_became_active":"2024-03-26T04:45:20.743351+0000","last_became_peered":"2024-03-26T04:45:20.743351+0000","last_unstale":"2024-03-26T04:46:00.469742+0000","last_undegraded":"2024-03-26T04:46:00.469742+0000","last_fullsized":"2024-03-26T04:46:00.469742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:41.152873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'128","reported_seq":275,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:02.181048+0000","last_change":"2024-03-26T04:45:10.297709+0000","last_active":"2024-03-26T04:46:02.181048+0000","last_peered":"2024-03-26T04:46:02.181048+0000","last_clean":"2024-03-26T04:46:02.181048+0000","last_became_active":"2024-03-26T04:45:10.297289+0000","last_became_peered":"2024-03-26T04:45:10.297289+0000","last_unstale":"2024-03-26T04:46:02.181048+0000","last_undegraded":"2024-03-26T04:46:02.181048+0000","last_fullsized":"2024-03-26T04:46:02.181048+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:43:00.973945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:43:00.973945+0000","last_clean_scrub_stamp":"2024-03-26T04:43:00.973945+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:59:45.901811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805973+0000","last_change":"2024-03-26T04:45:22.755358+0000","last_active":"2024-03-26T04:46:30.805973+0000","last_peered":"2024-03-26T04:46:30.805973+0000","last_clean":"2024-03-26T04:46:30.805973+0000","last_became_active":"2024-03-26T04:45:22.754994+0000","last_became_peered":"2024-03-26T04:45:22.754994+0000","last_unstale":"2024-03-26T04:46:30.805973+0000","last_undegraded":"2024-03-26T04:46:30.805973+0000","last_fullsized":"2024-03-26T04:46:30.805973+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:12:22.098046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470857+0000","last_change":"2024-03-26T04:45:24.779972+0000","last_active":"2024-03-26T04:46:00.470857+0000","last_peered":"2024-03-26T04:46:00.470857+0000","last_clean":"2024-03-26T04:46:00.470857+0000","last_became_active":"2024-03-26T04:45:24.779640+0000","last_became_peered":"2024-03-26T04:45:24.779640+0000","last_unstale":"2024-03-26T04:46:00.470857+0000","last_undegraded":"2024-03-26T04:46:00.470857+0000","last_fullsized":"2024-03-26T04:46:00.470857+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:52:16.161546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073679+0000","last_change":"2024-03-26T04:45:26.786569+0000","last_active":"2024-03-26T04:46:00.073679+0000","last_peered":"2024-03-26T04:46:00.073679+0000","last_clean":"2024-03-26T04:46:00.073679+0000","last_became_active":"2024-03-26T04:45:26.786242+0000","last_became_peered":"2024-03-26T04:45:26.786242+0000","last_unstale":"2024-03-26T04:46:00.073679+0000","last_undegraded":"2024-03-26T04:46:00.073679+0000","last_fullsized":"2024-03-26T04:46:00.073679+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:45:13.961941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199169+0000","last_change":"2024-03-26T04:45:24.770350+0000","last_active":"2024-03-26T04:46:00.199169+0000","last_peered":"2024-03-26T04:46:00.199169+0000","last_clean":"2024-03-26T04:46:00.199169+0000","last_became_active":"2024-03-26T04:45:20.745164+0000","last_became_peered":"2024-03-26T04:45:20.745164+0000","last_unstale":"2024-03-26T04:46:00.199169+0000","last_undegraded":"2024-03-26T04:46:00.199169+0000","last_fullsized":"2024-03-26T04:46:00.199169+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:19:45.864436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00118863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469274+0000","last_change":"2024-03-26T04:45:20.749345+0000","last_active":"2024-03-26T04:46:00.469274+0000","last_peered":"2024-03-26T04:46:00.469274+0000","last_clean":"2024-03-26T04:46:00.469274+0000","last_became_active":"2024-03-26T04:45:20.749063+0000","last_became_peered":"2024-03-26T04:45:20.749063+0000","last_unstale":"2024-03-26T04:46:00.469274+0000","last_undegraded":"2024-03-26T04:46:00.469274+0000","last_fullsized":"2024-03-26T04:46:00.469274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:01:32.095852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199744+0000","last_change":"2024-03-26T04:45:22.754658+0000","last_active":"2024-03-26T04:46:00.199744+0000","last_peered":"2024-03-26T04:46:00.199744+0000","last_clean":"2024-03-26T04:46:00.199744+0000","last_became_active":"2024-03-26T04:45:22.754226+0000","last_became_peered":"2024-03-26T04:45:22.754226+0000","last_unstale":"2024-03-26T04:46:00.199744+0000","last_undegraded":"2024-03-26T04:46:00.199744+0000","last_fullsized":"2024-03-26T04:46:00.199744+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:26:17.360139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470819+0000","last_change":"2024-03-26T04:45:24.783225+0000","last_active":"2024-03-26T04:46:00.470819+0000","last_peered":"2024-03-26T04:46:00.470819+0000","last_clean":"2024-03-26T04:46:00.470819+0000","last_became_active":"2024-03-26T04:45:24.782885+0000","last_became_peered":"2024-03-26T04:45:24.782885+0000","last_unstale":"2024-03-26T04:46:00.470819+0000","last_undegraded":"2024-03-26T04:46:00.470819+0000","last_fullsized":"2024-03-26T04:46:00.470819+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:49:49.195040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073133+0000","last_change":"2024-03-26T04:45:26.784002+0000","last_active":"2024-03-26T04:46:00.073133+0000","last_peered":"2024-03-26T04:46:00.073133+0000","last_clean":"2024-03-26T04:46:00.073133+0000","last_became_active":"2024-03-26T04:45:26.783676+0000","last_became_peered":"2024-03-26T04:45:26.783676+0000","last_unstale":"2024-03-26T04:46:00.073133+0000","last_undegraded":"2024-03-26T04:46:00.073133+0000","last_fullsized":"2024-03-26T04:46:00.073133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:19.711511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076344+0000","last_change":"2024-03-26T04:45:45.906497+0000","last_active":"2024-03-26T04:46:00.076344+0000","last_peered":"2024-03-26T04:46:00.076344+0000","last_clean":"2024-03-26T04:46:00.076344+0000","last_became_active":"2024-03-26T04:45:45.906096+0000","last_became_peered":"2024-03-26T04:45:45.906096+0000","last_unstale":"2024-03-26T04:46:00.076344+0000","last_undegraded":"2024-03-26T04:46:00.076344+0000","last_fullsized":"2024-03-26T04:46:00.076344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:21:38.012449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199206+0000","last_change":"2024-03-26T04:45:22.749002+0000","last_active":"2024-03-26T04:46:00.199206+0000","last_peered":"2024-03-26T04:46:00.199206+0000","last_clean":"2024-03-26T04:46:00.199206+0000","last_became_active":"2024-03-26T04:45:22.748634+0000","last_became_peered":"2024-03-26T04:45:22.748634+0000","last_unstale":"2024-03-26T04:46:00.199206+0000","last_undegraded":"2024-03-26T04:46:00.199206+0000","last_fullsized":"2024-03-26T04:46:00.199206+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:08:27.067377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470905+0000","last_change":"2024-03-26T04:45:24.775915+0000","last_active":"2024-03-26T04:46:00.470905+0000","last_peered":"2024-03-26T04:46:00.470905+0000","last_clean":"2024-03-26T04:46:00.470905+0000","last_became_active":"2024-03-26T04:45:24.775605+0000","last_became_peered":"2024-03-26T04:45:24.775605+0000","last_unstale":"2024-03-26T04:46:00.470905+0000","last_undegraded":"2024-03-26T04:46:00.470905+0000","last_fullsized":"2024-03-26T04:46:00.470905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:02:03.768569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075606+0000","last_change":"2024-03-26T04:45:26.781707+0000","last_active":"2024-03-26T04:46:00.075606+0000","last_peered":"2024-03-26T04:46:00.075606+0000","last_clean":"2024-03-26T04:46:00.075606+0000","last_became_active":"2024-03-26T04:45:26.781390+0000","last_became_peered":"2024-03-26T04:45:26.781390+0000","last_unstale":"2024-03-26T04:46:00.075606+0000","last_undegraded":"2024-03-26T04:46:00.075606+0000","last_fullsized":"2024-03-26T04:46:00.075606+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:37.471944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075624+0000","last_change":"2024-03-26T04:45:20.741148+0000","last_active":"2024-03-26T04:46:00.075624+0000","last_peered":"2024-03-26T04:46:00.075624+0000","last_clean":"2024-03-26T04:46:00.075624+0000","last_became_active":"2024-03-26T04:45:20.740818+0000","last_became_peered":"2024-03-26T04:45:20.740818+0000","last_unstale":"2024-03-26T04:46:00.075624+0000","last_undegraded":"2024-03-26T04:46:00.075624+0000","last_fullsized":"2024-03-26T04:46:00.075624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:51.332314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'30","reported_seq":85,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188166+0000","last_change":"2024-03-26T04:45:22.754301+0000","last_active":"2024-03-26T04:46:00.188166+0000","last_peered":"2024-03-26T04:46:00.188166+0000","last_clean":"2024-03-26T04:46:00.188166+0000","last_became_active":"2024-03-26T04:45:22.753895+0000","last_became_peered":"2024-03-26T04:45:22.753895+0000","last_unstale":"2024-03-26T04:46:00.188166+0000","last_undegraded":"2024-03-26T04:46:00.188166+0000","last_fullsized":"2024-03-26T04:46:00.188166+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:05:51.617244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075299+0000","last_change":"2024-03-26T04:45:24.774608+0000","last_active":"2024-03-26T04:46:00.075299+0000","last_peered":"2024-03-26T04:46:00.075299+0000","last_clean":"2024-03-26T04:46:00.075299+0000","last_became_active":"2024-03-26T04:45:24.773823+0000","last_became_peered":"2024-03-26T04:45:24.773823+0000","last_unstale":"2024-03-26T04:46:00.075299+0000","last_undegraded":"2024-03-26T04:46:00.075299+0000","last_fullsized":"2024-03-26T04:46:00.075299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:35.588693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470355+0000","last_change":"2024-03-26T04:45:26.786034+0000","last_active":"2024-03-26T04:46:00.470355+0000","last_peered":"2024-03-26T04:46:00.470355+0000","last_clean":"2024-03-26T04:46:00.470355+0000","last_became_active":"2024-03-26T04:45:26.785734+0000","last_became_peered":"2024-03-26T04:45:26.785734+0000","last_unstale":"2024-03-26T04:46:00.470355+0000","last_undegraded":"2024-03-26T04:46:00.470355+0000","last_fullsized":"2024-03-26T04:46:00.470355+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:51:38.718687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470957+0000","last_change":"2024-03-26T04:45:20.738453+0000","last_active":"2024-03-26T04:46:00.470957+0000","last_peered":"2024-03-26T04:46:00.470957+0000","last_clean":"2024-03-26T04:46:00.470957+0000","last_became_active":"2024-03-26T04:45:20.738132+0000","last_became_peered":"2024-03-26T04:45:20.738132+0000","last_unstale":"2024-03-26T04:46:00.470957+0000","last_undegraded":"2024-03-26T04:46:00.470957+0000","last_fullsized":"2024-03-26T04:46:00.470957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:20.826382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075156+0000","last_change":"2024-03-26T04:45:22.754771+0000","last_active":"2024-03-26T04:46:00.075156+0000","last_peered":"2024-03-26T04:46:00.075156+0000","last_clean":"2024-03-26T04:46:00.075156+0000","last_became_active":"2024-03-26T04:45:22.754436+0000","last_became_peered":"2024-03-26T04:45:22.754436+0000","last_unstale":"2024-03-26T04:46:00.075156+0000","last_undegraded":"2024-03-26T04:46:00.075156+0000","last_fullsized":"2024-03-26T04:46:00.075156+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:56.003046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805821+0000","last_change":"2024-03-26T04:45:24.778428+0000","last_active":"2024-03-26T04:46:30.805821+0000","last_peered":"2024-03-26T04:46:30.805821+0000","last_clean":"2024-03-26T04:46:30.805821+0000","last_became_active":"2024-03-26T04:45:24.778015+0000","last_became_peered":"2024-03-26T04:45:24.778015+0000","last_unstale":"2024-03-26T04:46:30.805821+0000","last_undegraded":"2024-03-26T04:46:30.805821+0000","last_fullsized":"2024-03-26T04:46:30.805821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:39.901802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188946+0000","last_change":"2024-03-26T04:45:26.786936+0000","last_active":"2024-03-26T04:46:00.188946+0000","last_peered":"2024-03-26T04:46:00.188946+0000","last_clean":"2024-03-26T04:46:00.188946+0000","last_became_active":"2024-03-26T04:45:26.786635+0000","last_became_peered":"2024-03-26T04:45:26.786635+0000","last_unstale":"2024-03-26T04:46:00.188946+0000","last_undegraded":"2024-03-26T04:46:00.188946+0000","last_fullsized":"2024-03-26T04:46:00.188946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:30.058065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470665+0000","last_change":"2024-03-26T04:45:24.778189+0000","last_active":"2024-03-26T04:46:00.470665+0000","last_peered":"2024-03-26T04:46:00.470665+0000","last_clean":"2024-03-26T04:46:00.470665+0000","last_became_active":"2024-03-26T04:45:24.777853+0000","last_became_peered":"2024-03-26T04:45:24.777853+0000","last_unstale":"2024-03-26T04:46:00.470665+0000","last_undegraded":"2024-03-26T04:46:00.470665+0000","last_fullsized":"2024-03-26T04:46:00.470665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:07:50.814289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469477+0000","last_change":"2024-03-26T04:45:20.749005+0000","last_active":"2024-03-26T04:46:00.469477+0000","last_peered":"2024-03-26T04:46:00.469477+0000","last_clean":"2024-03-26T04:46:00.469477+0000","last_became_active":"2024-03-26T04:45:20.748662+0000","last_became_peered":"2024-03-26T04:45:20.748662+0000","last_unstale":"2024-03-26T04:46:00.469477+0000","last_undegraded":"2024-03-26T04:46:00.469477+0000","last_fullsized":"2024-03-26T04:46:00.469477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:12:17.013354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076189+0000","last_change":"2024-03-26T04:45:22.756409+0000","last_active":"2024-03-26T04:46:00.076189+0000","last_peered":"2024-03-26T04:46:00.076189+0000","last_clean":"2024-03-26T04:46:00.076189+0000","last_became_active":"2024-03-26T04:45:22.756082+0000","last_became_peered":"2024-03-26T04:45:22.756082+0000","last_unstale":"2024-03-26T04:46:00.076189+0000","last_undegraded":"2024-03-26T04:46:00.076189+0000","last_fullsized":"2024-03-26T04:46:00.076189+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:38:09.542742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471573+0000","last_change":"2024-03-26T04:45:26.790427+0000","last_active":"2024-03-26T04:46:00.471573+0000","last_peered":"2024-03-26T04:46:00.471573+0000","last_clean":"2024-03-26T04:46:00.471573+0000","last_became_active":"2024-03-26T04:45:26.789473+0000","last_became_peered":"2024-03-26T04:45:26.789473+0000","last_unstale":"2024-03-26T04:46:00.471573+0000","last_undegraded":"2024-03-26T04:46:00.471573+0000","last_fullsized":"2024-03-26T04:46:00.471573+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:39.960035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805798+0000","last_change":"2024-03-26T04:45:24.776004+0000","last_active":"2024-03-26T04:46:30.805798+0000","last_peered":"2024-03-26T04:46:30.805798+0000","last_clean":"2024-03-26T04:46:30.805798+0000","last_became_active":"2024-03-26T04:45:24.775501+0000","last_became_peered":"2024-03-26T04:45:24.775501+0000","last_unstale":"2024-03-26T04:46:30.805798+0000","last_undegraded":"2024-03-26T04:46:30.805798+0000","last_fullsized":"2024-03-26T04:46:30.805798+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:20:17.582384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073429+0000","last_change":"2024-03-26T04:45:20.738821+0000","last_active":"2024-03-26T04:46:00.073429+0000","last_peered":"2024-03-26T04:46:00.073429+0000","last_clean":"2024-03-26T04:46:00.073429+0000","last_became_active":"2024-03-26T04:45:20.737911+0000","last_became_peered":"2024-03-26T04:45:20.737911+0000","last_unstale":"2024-03-26T04:46:00.073429+0000","last_undegraded":"2024-03-26T04:46:00.073429+0000","last_fullsized":"2024-03-26T04:46:00.073429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:26:37.391492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074225+0000","last_change":"2024-03-26T04:45:22.760530+0000","last_active":"2024-03-26T04:46:00.074225+0000","last_peered":"2024-03-26T04:46:00.074225+0000","last_clean":"2024-03-26T04:46:00.074225+0000","last_became_active":"2024-03-26T04:45:22.760214+0000","last_became_peered":"2024-03-26T04:45:22.760214+0000","last_unstale":"2024-03-26T04:46:00.074225+0000","last_undegraded":"2024-03-26T04:46:00.074225+0000","last_fullsized":"2024-03-26T04:46:00.074225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:25.721050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076005+0000","last_change":"2024-03-26T04:45:26.776596+0000","last_active":"2024-03-26T04:46:00.076005+0000","last_peered":"2024-03-26T04:46:00.076005+0000","last_clean":"2024-03-26T04:46:00.076005+0000","last_became_active":"2024-03-26T04:45:26.776303+0000","last_became_peered":"2024-03-26T04:45:26.776303+0000","last_unstale":"2024-03-26T04:46:00.076005+0000","last_undegraded":"2024-03-26T04:46:00.076005+0000","last_fullsized":"2024-03-26T04:46:00.076005+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:18:15.114703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805494+0000","last_change":"2024-03-26T04:45:24.776804+0000","last_active":"2024-03-26T04:46:30.805494+0000","last_peered":"2024-03-26T04:46:30.805494+0000","last_clean":"2024-03-26T04:46:30.805494+0000","last_became_active":"2024-03-26T04:45:24.776464+0000","last_became_peered":"2024-03-26T04:45:24.776464+0000","last_unstale":"2024-03-26T04:46:30.805494+0000","last_undegraded":"2024-03-26T04:46:30.805494+0000","last_fullsized":"2024-03-26T04:46:30.805494+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:47:49.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075022+0000","last_change":"2024-03-26T04:45:20.741586+0000","last_active":"2024-03-26T04:46:00.075022+0000","last_peered":"2024-03-26T04:46:00.075022+0000","last_clean":"2024-03-26T04:46:00.075022+0000","last_became_active":"2024-03-26T04:45:20.741234+0000","last_became_peered":"2024-03-26T04:45:20.741234+0000","last_unstale":"2024-03-26T04:46:00.075022+0000","last_undegraded":"2024-03-26T04:46:00.075022+0000","last_fullsized":"2024-03-26T04:46:00.075022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:27:24.311362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075357+0000","last_change":"2024-03-26T04:45:22.759704+0000","last_active":"2024-03-26T04:46:00.075357+0000","last_peered":"2024-03-26T04:46:00.075357+0000","last_clean":"2024-03-26T04:46:00.075357+0000","last_became_active":"2024-03-26T04:45:22.759452+0000","last_became_peered":"2024-03-26T04:45:22.759452+0000","last_unstale":"2024-03-26T04:46:00.075357+0000","last_undegraded":"2024-03-26T04:46:00.075357+0000","last_fullsized":"2024-03-26T04:46:00.075357+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:00.756040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076887+0000","last_change":"2024-03-26T04:45:26.784724+0000","last_active":"2024-03-26T04:46:00.076887+0000","last_peered":"2024-03-26T04:46:00.076887+0000","last_clean":"2024-03-26T04:46:00.076887+0000","last_became_active":"2024-03-26T04:45:26.784375+0000","last_became_peered":"2024-03-26T04:45:26.784375+0000","last_unstale":"2024-03-26T04:46:00.076887+0000","last_undegraded":"2024-03-26T04:46:00.076887+0000","last_fullsized":"2024-03-26T04:46:00.076887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:19:38.728539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073771+0000","last_change":"2024-03-26T04:45:24.772992+0000","last_active":"2024-03-26T04:46:00.073771+0000","last_peered":"2024-03-26T04:46:00.073771+0000","last_clean":"2024-03-26T04:46:00.073771+0000","last_became_active":"2024-03-26T04:45:24.772653+0000","last_became_peered":"2024-03-26T04:45:24.772653+0000","last_unstale":"2024-03-26T04:46:00.073771+0000","last_undegraded":"2024-03-26T04:46:00.073771+0000","last_fullsized":"2024-03-26T04:46:00.073771+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:38.757217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469969+0000","last_change":"2024-03-26T04:45:20.743969+0000","last_active":"2024-03-26T04:46:00.469969+0000","last_peered":"2024-03-26T04:46:00.469969+0000","last_clean":"2024-03-26T04:46:00.469969+0000","last_became_active":"2024-03-26T04:45:20.743666+0000","last_became_peered":"2024-03-26T04:45:20.743666+0000","last_unstale":"2024-03-26T04:46:00.469969+0000","last_undegraded":"2024-03-26T04:46:00.469969+0000","last_fullsized":"2024-03-26T04:46:00.469969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:26:37.306783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.077244+0000","last_change":"2024-03-26T04:45:45.914102+0000","last_active":"2024-03-26T04:46:00.077244+0000","last_peered":"2024-03-26T04:46:00.077244+0000","last_clean":"2024-03-26T04:46:00.077244+0000","last_became_active":"2024-03-26T04:45:45.905675+0000","last_became_peered":"2024-03-26T04:45:45.905675+0000","last_unstale":"2024-03-26T04:46:00.077244+0000","last_undegraded":"2024-03-26T04:46:00.077244+0000","last_fullsized":"2024-03-26T04:46:00.077244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:27:08.036919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471195+0000","last_change":"2024-03-26T04:45:26.783776+0000","last_active":"2024-03-26T04:46:00.471195+0000","last_peered":"2024-03-26T04:46:00.471195+0000","last_clean":"2024-03-26T04:46:00.471195+0000","last_became_active":"2024-03-26T04:45:26.783416+0000","last_became_peered":"2024-03-26T04:45:26.783416+0000","last_unstale":"2024-03-26T04:46:00.471195+0000","last_undegraded":"2024-03-26T04:46:00.471195+0000","last_fullsized":"2024-03-26T04:46:00.471195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:06:04.970363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076316+0000","last_change":"2024-03-26T04:45:24.777500+0000","last_active":"2024-03-26T04:46:00.076316+0000","last_peered":"2024-03-26T04:46:00.076316+0000","last_clean":"2024-03-26T04:46:00.076316+0000","last_became_active":"2024-03-26T04:45:24.777177+0000","last_became_peered":"2024-03-26T04:45:24.777177+0000","last_unstale":"2024-03-26T04:46:00.076316+0000","last_undegraded":"2024-03-26T04:46:00.076316+0000","last_fullsized":"2024-03-26T04:46:00.076316+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:45:33.144903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076617+0000","last_change":"2024-03-26T04:45:20.738638+0000","last_active":"2024-03-26T04:46:00.076617+0000","last_peered":"2024-03-26T04:46:00.076617+0000","last_clean":"2024-03-26T04:46:00.076617+0000","last_became_active":"2024-03-26T04:45:20.738332+0000","last_became_peered":"2024-03-26T04:45:20.738332+0000","last_unstale":"2024-03-26T04:46:00.076617+0000","last_undegraded":"2024-03-26T04:46:00.076617+0000","last_fullsized":"2024-03-26T04:46:00.076617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:53:00.385609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470652+0000","last_change":"2024-03-26T04:45:22.758745+0000","last_active":"2024-03-26T04:46:00.470652+0000","last_peered":"2024-03-26T04:46:00.470652+0000","last_clean":"2024-03-26T04:46:00.470652+0000","last_became_active":"2024-03-26T04:45:22.758473+0000","last_became_peered":"2024-03-26T04:45:22.758473+0000","last_unstale":"2024-03-26T04:46:00.470652+0000","last_undegraded":"2024-03-26T04:46:00.470652+0000","last_fullsized":"2024-03-26T04:46:00.470652+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:18.388864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.198866+0000","last_change":"2024-03-26T04:45:26.788455+0000","last_active":"2024-03-26T04:46:00.198866+0000","last_peered":"2024-03-26T04:46:00.198866+0000","last_clean":"2024-03-26T04:46:00.198866+0000","last_became_active":"2024-03-26T04:45:26.788123+0000","last_became_peered":"2024-03-26T04:45:26.788123+0000","last_unstale":"2024-03-26T04:46:00.198866+0000","last_undegraded":"2024-03-26T04:46:00.198866+0000","last_fullsized":"2024-03-26T04:46:00.198866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:10.716106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076443+0000","last_change":"2024-03-26T04:45:24.773439+0000","last_active":"2024-03-26T04:46:00.076443+0000","last_peered":"2024-03-26T04:46:00.076443+0000","last_clean":"2024-03-26T04:46:00.076443+0000","last_became_active":"2024-03-26T04:45:24.773079+0000","last_became_peered":"2024-03-26T04:45:24.773079+0000","last_unstale":"2024-03-26T04:46:00.076443+0000","last_undegraded":"2024-03-26T04:46:00.076443+0000","last_fullsized":"2024-03-26T04:46:00.076443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:55:08.823253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074080+0000","last_change":"2024-03-26T04:45:20.738319+0000","last_active":"2024-03-26T04:46:00.074080+0000","last_peered":"2024-03-26T04:46:00.074080+0000","last_clean":"2024-03-26T04:46:00.074080+0000","last_became_active":"2024-03-26T04:45:20.737734+0000","last_became_peered":"2024-03-26T04:45:20.737734+0000","last_unstale":"2024-03-26T04:46:00.074080+0000","last_undegraded":"2024-03-26T04:46:00.074080+0000","last_fullsized":"2024-03-26T04:46:00.074080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:34:03.766481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199530+0000","last_change":"2024-03-26T04:45:22.756790+0000","last_active":"2024-03-26T04:46:00.199530+0000","last_peered":"2024-03-26T04:46:00.199530+0000","last_clean":"2024-03-26T04:46:00.199530+0000","last_became_active":"2024-03-26T04:45:22.756463+0000","last_became_peered":"2024-03-26T04:45:22.756463+0000","last_unstale":"2024-03-26T04:46:00.199530+0000","last_undegraded":"2024-03-26T04:46:00.199530+0000","last_fullsized":"2024-03-26T04:46:00.199530+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:55:08.853240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469287+0000","last_change":"2024-03-26T04:45:26.790222+0000","last_active":"2024-03-26T04:46:00.469287+0000","last_peered":"2024-03-26T04:46:00.469287+0000","last_clean":"2024-03-26T04:46:00.469287+0000","last_became_active":"2024-03-26T04:45:26.789912+0000","last_became_peered":"2024-03-26T04:45:26.789912+0000","last_unstale":"2024-03-26T04:46:00.469287+0000","last_undegraded":"2024-03-26T04:46:00.469287+0000","last_fullsized":"2024-03-26T04:46:00.469287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:43.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076687+0000","last_change":"2024-03-26T04:45:20.735677+0000","last_active":"2024-03-26T04:46:00.076687+0000","last_peered":"2024-03-26T04:46:00.076687+0000","last_clean":"2024-03-26T04:46:00.076687+0000","last_became_active":"2024-03-26T04:45:20.735355+0000","last_became_peered":"2024-03-26T04:45:20.735355+0000","last_unstale":"2024-03-26T04:46:00.076687+0000","last_undegraded":"2024-03-26T04:46:00.076687+0000","last_fullsized":"2024-03-26T04:46:00.076687+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:38:31.372686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'27","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469398+0000","last_change":"2024-03-26T04:45:22.752501+0000","last_active":"2024-03-26T04:46:00.469398+0000","last_peered":"2024-03-26T04:46:00.469398+0000","last_clean":"2024-03-26T04:46:00.469398+0000","last_became_active":"2024-03-26T04:45:22.752191+0000","last_became_peered":"2024-03-26T04:45:22.752191+0000","last_unstale":"2024-03-26T04:46:00.469398+0000","last_undegraded":"2024-03-26T04:46:00.469398+0000","last_fullsized":"2024-03-26T04:46:00.469398+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:16:49.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:30.805841+0000","last_change":"2024-03-26T04:45:24.780983+0000","last_active":"2024-03-26T04:46:30.805841+0000","last_peered":"2024-03-26T04:46:30.805841+0000","last_clean":"2024-03-26T04:46:30.805841+0000","last_became_active":"2024-03-26T04:45:24.780415+0000","last_became_peered":"2024-03-26T04:45:24.780415+0000","last_unstale":"2024-03-26T04:46:30.805841+0000","last_undegraded":"2024-03-26T04:46:30.805841+0000","last_fullsized":"2024-03-26T04:46:30.805841+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:49:30.547179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":65,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32540,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712612,"statfs":{"total":95995035648,"available":95961714688,"internally_reserved":0,"allocated":1392640,"data_stored":923704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1380352,"data_stored":922634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":811008,"data_stored":333720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1400832,"data_stored":923844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":794624,"data_stored":332839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":823296,"data_stored":333106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":798720,"data_stored":332344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":786432,"data_stored":332619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:46:43.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:46:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[29863]: Deploying daemon grafana.a on smithi186 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:43 smithi012 ceph-mon[41141]: Deploying daemon grafana.a on smithi186 2024-03-26T04:46:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:43 smithi186 ceph-mon[33438]: Deploying daemon grafana.a on smithi186 2024-03-26T04:46:44.631 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:44 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:44.522Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000700121s 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 -- 172.21.15.12:0/3456845696 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c102a90 msgr2=0x7f472c10eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 --2- 172.21.15.12:0/3456845696 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c102a90 0x7f472c10eed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f472400b0a0 tx=0x7f472400b3b0 comp rx=0 tx=0).stop 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 -- 172.21.15.12:0/3456845696 shutdown_connections 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 --2- 172.21.15.12:0/3456845696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c1080b0 0x7f472c108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 --2- 172.21.15.12:0/3456845696 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c102a90 0x7f472c10eed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 --2- 172.21.15.12:0/3456845696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f472c1020d0 0x7f472c102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 -- 172.21.15.12:0/3456845696 >> 172.21.15.12:0/3456845696 conn(0x7f472c0791b0 msgr2=0x7f472c0795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 -- 172.21.15.12:0/3456845696 shutdown_connections 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.865+0000 7f473085f700 1 -- 172.21.15.12:0/3456845696 wait complete. 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 Processor -- start 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 -- start start 2024-03-26T04:46:44.876 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 0x7f472c19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c102a90 0x7f472c19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f472c1080b0 0x7f472c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c198020 con 0x7f472c102a90 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c198160 con 0x7f472c1080b0 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.866+0000 7f473085f700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c198320 con 0x7f472c1020d0 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 0x7f472c19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f4729d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c102a90 0x7f472c19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 0x7f472c19d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:42886/0 (socket says 172.21.15.12:42886) 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 -- 172.21.15.12:0/1369617662 learned_addr learned my addr 172.21.15.12:0/1369617662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472ad9d700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f472c1080b0 0x7f472c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 -- 172.21.15.12:0/1369617662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f472c1080b0 msgr2=0x7f472c1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f472c1080b0 0x7f472c1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 -- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c102a90 msgr2=0x7f472c19dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c102a90 0x7f472c19dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:44.877 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 -- 172.21.15.12:0/1369617662 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c009520 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f4729d9b700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c102a90 0x7f472c19dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f472a59c700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 0x7f472c19d7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f471c00eb30 tx=0x7f471c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c009880 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4724009d00 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f471c0044a0 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c1a4680 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.867+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c003c40 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.868+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c068d40 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.869+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f471c003dc0 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.869+0000 7f471b7fe700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f471406dac0 0x7f471406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.869+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f471c014070 con 0x7f472c1020d0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.869+0000 7f4729d9b700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f471406dac0 0x7f471406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.869+0000 7f4729d9b700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f471406dac0 0x7f471406ff80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f472c1121b0 tx=0x7f4720009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:44.878 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:44.874+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f471c057020 con 0x7f472c1020d0 2024-03-26T04:46:45.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.072+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 --> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f472c1115b0 con 0x7f471406dac0 2024-03-26T04:46:45.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.082+0000 7f471b7fe700 1 -- 172.21.15.12:0/1369617662 <== mgr.24464 v2:172.21.15.12:6800/715138906 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346508 (secure 0 0 0) 0x7f472c1115b0 con 0x7f471406dac0 2024-03-26T04:46:45.086 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f471406dac0 msgr2=0x7f471406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f471406dac0 0x7f471406ff80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f472c1121b0 tx=0x7f4720009260 comp rx=0 tx=0).stop 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 msgr2=0x7f472c19d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 0x7f472c19d7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f471c00eb30 tx=0x7f471c00ee40 comp rx=0 tx=0).stop 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 shutdown_connections 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f471406dac0 0x7f471406ff80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f472c1080b0 0x7f472c1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f472c1020d0 0x7f472c19d7b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 --2- 172.21.15.12:0/1369617662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f472c102a90 0x7f472c19dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 >> 172.21.15.12:0/1369617662 conn(0x7f472c0791b0 msgr2=0x7f472c105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:45.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 shutdown_connections 2024-03-26T04:46:45.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:45.085+0000 7f473085f700 1 -- 172.21.15.12:0/1369617662 wait complete. 2024-03-26T04:46:45.090 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-03-26T04:46:45.800 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-26T04:46:44.095958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1296,"num_read_kb":1110,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1771764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236628,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724588,"statfs":{"total":767960285184,"available":767717978112,"internally_reserved":0,"allocated":8187904,"data_stored":4434810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003514"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188910+0000","last_change":"2024-03-26T04:45:26.786492+0000","last_active":"2024-03-26T04:46:00.188910+0000","last_peered":"2024-03-26T04:46:00.188910+0000","last_clean":"2024-03-26T04:46:00.188910+0000","last_became_active":"2024-03-26T04:45:26.786094+0000","last_became_peered":"2024-03-26T04:45:26.786094+0000","last_unstale":"2024-03-26T04:46:00.188910+0000","last_undegraded":"2024-03-26T04:46:00.188910+0000","last_fullsized":"2024-03-26T04:46:00.188910+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:49.907754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074989+0000","last_change":"2024-03-26T04:45:24.771946+0000","last_active":"2024-03-26T04:46:00.074989+0000","last_peered":"2024-03-26T04:46:00.074989+0000","last_clean":"2024-03-26T04:46:00.074989+0000","last_became_active":"2024-03-26T04:45:24.771633+0000","last_became_peered":"2024-03-26T04:45:24.771633+0000","last_unstale":"2024-03-26T04:46:00.074989+0000","last_undegraded":"2024-03-26T04:46:00.074989+0000","last_fullsized":"2024-03-26T04:46:00.074989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:06:27.751659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199569+0000","last_change":"2024-03-26T04:45:20.750121+0000","last_active":"2024-03-26T04:46:00.199569+0000","last_peered":"2024-03-26T04:46:00.199569+0000","last_clean":"2024-03-26T04:46:00.199569+0000","last_became_active":"2024-03-26T04:45:20.749745+0000","last_became_peered":"2024-03-26T04:45:20.749745+0000","last_unstale":"2024-03-26T04:46:00.199569+0000","last_undegraded":"2024-03-26T04:46:00.199569+0000","last_fullsized":"2024-03-26T04:46:00.199569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:30:39.908300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"67'39","reported_seq":102,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188678+0000","last_change":"2024-03-26T04:45:22.754431+0000","last_active":"2024-03-26T04:46:00.188678+0000","last_peered":"2024-03-26T04:46:00.188678+0000","last_clean":"2024-03-26T04:46:00.188678+0000","last_became_active":"2024-03-26T04:45:22.754121+0000","last_became_peered":"2024-03-26T04:45:22.754121+0000","last_unstale":"2024-03-26T04:46:00.188678+0000","last_undegraded":"2024-03-26T04:46:00.188678+0000","last_fullsized":"2024-03-26T04:46:00.188678+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:18:36.681830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075201+0000","last_change":"2024-03-26T04:45:26.778088+0000","last_active":"2024-03-26T04:46:00.075201+0000","last_peered":"2024-03-26T04:46:00.075201+0000","last_clean":"2024-03-26T04:46:00.075201+0000","last_became_active":"2024-03-26T04:45:26.777655+0000","last_became_peered":"2024-03-26T04:45:26.777655+0000","last_unstale":"2024-03-26T04:46:00.075201+0000","last_undegraded":"2024-03-26T04:46:00.075201+0000","last_fullsized":"2024-03-26T04:46:00.075201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:03:57.686967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073263+0000","last_change":"2024-03-26T04:45:24.780540+0000","last_active":"2024-03-26T04:46:00.073263+0000","last_peered":"2024-03-26T04:46:00.073263+0000","last_clean":"2024-03-26T04:46:00.073263+0000","last_became_active":"2024-03-26T04:45:24.780166+0000","last_became_peered":"2024-03-26T04:45:24.780166+0000","last_unstale":"2024-03-26T04:46:00.073263+0000","last_undegraded":"2024-03-26T04:46:00.073263+0000","last_fullsized":"2024-03-26T04:46:00.073263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:15:33.027204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189809+0000","last_change":"2024-03-26T04:45:20.739973+0000","last_active":"2024-03-26T04:46:00.189809+0000","last_peered":"2024-03-26T04:46:00.189809+0000","last_clean":"2024-03-26T04:46:00.189809+0000","last_became_active":"2024-03-26T04:45:20.739684+0000","last_became_peered":"2024-03-26T04:45:20.739684+0000","last_unstale":"2024-03-26T04:46:00.189809+0000","last_undegraded":"2024-03-26T04:46:00.189809+0000","last_fullsized":"2024-03-26T04:46:00.189809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:51:56.871362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075133+0000","last_change":"2024-03-26T04:45:22.756717+0000","last_active":"2024-03-26T04:46:00.075133+0000","last_peered":"2024-03-26T04:46:00.075133+0000","last_clean":"2024-03-26T04:46:00.075133+0000","last_became_active":"2024-03-26T04:45:22.756401+0000","last_became_peered":"2024-03-26T04:45:22.756401+0000","last_unstale":"2024-03-26T04:46:00.075133+0000","last_undegraded":"2024-03-26T04:46:00.075133+0000","last_fullsized":"2024-03-26T04:46:00.075133+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:06:03.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469636+0000","last_change":"2024-03-26T04:45:26.785043+0000","last_active":"2024-03-26T04:46:00.469636+0000","last_peered":"2024-03-26T04:46:00.469636+0000","last_clean":"2024-03-26T04:46:00.469636+0000","last_became_active":"2024-03-26T04:45:26.784729+0000","last_became_peered":"2024-03-26T04:45:26.784729+0000","last_unstale":"2024-03-26T04:46:00.469636+0000","last_undegraded":"2024-03-26T04:46:00.469636+0000","last_fullsized":"2024-03-26T04:46:00.469636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:12:03.696267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470488+0000","last_change":"2024-03-26T04:45:24.779587+0000","last_active":"2024-03-26T04:46:00.470488+0000","last_peered":"2024-03-26T04:46:00.470488+0000","last_clean":"2024-03-26T04:46:00.470488+0000","last_became_active":"2024-03-26T04:45:24.779269+0000","last_became_peered":"2024-03-26T04:45:24.779269+0000","last_unstale":"2024-03-26T04:46:00.470488+0000","last_undegraded":"2024-03-26T04:46:00.470488+0000","last_fullsized":"2024-03-26T04:46:00.470488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:08:44.536805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471032+0000","last_change":"2024-03-26T04:45:20.750226+0000","last_active":"2024-03-26T04:46:00.471032+0000","last_peered":"2024-03-26T04:46:00.471032+0000","last_clean":"2024-03-26T04:46:00.471032+0000","last_became_active":"2024-03-26T04:45:20.749958+0000","last_became_peered":"2024-03-26T04:45:20.749958+0000","last_unstale":"2024-03-26T04:46:00.471032+0000","last_undegraded":"2024-03-26T04:46:00.471032+0000","last_fullsized":"2024-03-26T04:46:00.471032+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:15:34.043794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075703+0000","last_change":"2024-03-26T04:45:22.757451+0000","last_active":"2024-03-26T04:46:00.075703+0000","last_peered":"2024-03-26T04:46:00.075703+0000","last_clean":"2024-03-26T04:46:00.075703+0000","last_became_active":"2024-03-26T04:45:22.757121+0000","last_became_peered":"2024-03-26T04:45:22.757121+0000","last_unstale":"2024-03-26T04:46:00.075703+0000","last_undegraded":"2024-03-26T04:46:00.075703+0000","last_fullsized":"2024-03-26T04:46:00.075703+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:43:14.048606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199206+0000","last_change":"2024-03-26T04:45:26.788470+0000","last_active":"2024-03-26T04:46:00.199206+0000","last_peered":"2024-03-26T04:46:00.199206+0000","last_clean":"2024-03-26T04:46:00.199206+0000","last_became_active":"2024-03-26T04:45:26.788138+0000","last_became_peered":"2024-03-26T04:45:26.788138+0000","last_unstale":"2024-03-26T04:46:00.199206+0000","last_undegraded":"2024-03-26T04:46:00.199206+0000","last_fullsized":"2024-03-26T04:46:00.199206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:24:44.757867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469833+0000","last_change":"2024-03-26T04:45:24.777381+0000","last_active":"2024-03-26T04:46:00.469833+0000","last_peered":"2024-03-26T04:46:00.469833+0000","last_clean":"2024-03-26T04:46:00.469833+0000","last_became_active":"2024-03-26T04:45:24.777084+0000","last_became_peered":"2024-03-26T04:45:24.777084+0000","last_unstale":"2024-03-26T04:46:00.469833+0000","last_undegraded":"2024-03-26T04:46:00.469833+0000","last_fullsized":"2024-03-26T04:46:00.469833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:54:36.974749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470644+0000","last_change":"2024-03-26T04:45:20.740965+0000","last_active":"2024-03-26T04:46:00.470644+0000","last_peered":"2024-03-26T04:46:00.470644+0000","last_clean":"2024-03-26T04:46:00.470644+0000","last_became_active":"2024-03-26T04:45:20.740678+0000","last_became_peered":"2024-03-26T04:45:20.740678+0000","last_unstale":"2024-03-26T04:46:00.470644+0000","last_undegraded":"2024-03-26T04:46:00.470644+0000","last_fullsized":"2024-03-26T04:46:00.470644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:09:57.474208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075302+0000","last_change":"2024-03-26T04:45:22.755814+0000","last_active":"2024-03-26T04:46:00.075302+0000","last_peered":"2024-03-26T04:46:00.075302+0000","last_clean":"2024-03-26T04:46:00.075302+0000","last_became_active":"2024-03-26T04:45:22.755475+0000","last_became_peered":"2024-03-26T04:45:22.755475+0000","last_unstale":"2024-03-26T04:46:00.075302+0000","last_undegraded":"2024-03-26T04:46:00.075302+0000","last_fullsized":"2024-03-26T04:46:00.075302+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:35:21.930949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189890+0000","last_change":"2024-03-26T04:45:26.788091+0000","last_active":"2024-03-26T04:46:00.189890+0000","last_peered":"2024-03-26T04:46:00.189890+0000","last_clean":"2024-03-26T04:46:00.189890+0000","last_became_active":"2024-03-26T04:45:26.787742+0000","last_became_peered":"2024-03-26T04:45:26.787742+0000","last_unstale":"2024-03-26T04:46:00.189890+0000","last_undegraded":"2024-03-26T04:46:00.189890+0000","last_fullsized":"2024-03-26T04:46:00.189890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:04:19.783833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075495+0000","last_change":"2024-03-26T04:45:24.775474+0000","last_active":"2024-03-26T04:46:00.075495+0000","last_peered":"2024-03-26T04:46:00.075495+0000","last_clean":"2024-03-26T04:46:00.075495+0000","last_became_active":"2024-03-26T04:45:24.775070+0000","last_became_peered":"2024-03-26T04:45:24.775070+0000","last_unstale":"2024-03-26T04:46:00.075495+0000","last_undegraded":"2024-03-26T04:46:00.075495+0000","last_fullsized":"2024-03-26T04:46:00.075495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:08:56.589055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189092+0000","last_change":"2024-03-26T04:45:20.738137+0000","last_active":"2024-03-26T04:46:00.189092+0000","last_peered":"2024-03-26T04:46:00.189092+0000","last_clean":"2024-03-26T04:46:00.189092+0000","last_became_active":"2024-03-26T04:45:20.737112+0000","last_became_peered":"2024-03-26T04:45:20.737112+0000","last_unstale":"2024-03-26T04:46:00.189092+0000","last_undegraded":"2024-03-26T04:46:00.189092+0000","last_fullsized":"2024-03-26T04:46:00.189092+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:09:18.767305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":84,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189635+0000","last_change":"2024-03-26T04:45:45.921661+0000","last_active":"2024-03-26T04:46:00.189635+0000","last_peered":"2024-03-26T04:46:00.189635+0000","last_clean":"2024-03-26T04:46:00.189635+0000","last_became_active":"2024-03-26T04:45:45.906481+0000","last_became_peered":"2024-03-26T04:45:45.906481+0000","last_unstale":"2024-03-26T04:46:00.189635+0000","last_undegraded":"2024-03-26T04:46:00.189635+0000","last_fullsized":"2024-03-26T04:46:00.189635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:15:47.485396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076166+0000","last_change":"2024-03-26T04:45:26.782142+0000","last_active":"2024-03-26T04:46:00.076166+0000","last_peered":"2024-03-26T04:46:00.076166+0000","last_clean":"2024-03-26T04:46:00.076166+0000","last_became_active":"2024-03-26T04:45:26.781832+0000","last_became_peered":"2024-03-26T04:45:26.781832+0000","last_unstale":"2024-03-26T04:46:00.076166+0000","last_undegraded":"2024-03-26T04:46:00.076166+0000","last_fullsized":"2024-03-26T04:46:00.076166+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:52:01.699447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074168+0000","last_change":"2024-03-26T04:45:24.771980+0000","last_active":"2024-03-26T04:46:00.074168+0000","last_peered":"2024-03-26T04:46:00.074168+0000","last_clean":"2024-03-26T04:46:00.074168+0000","last_became_active":"2024-03-26T04:45:24.771665+0000","last_became_peered":"2024-03-26T04:45:24.771665+0000","last_unstale":"2024-03-26T04:46:00.074168+0000","last_undegraded":"2024-03-26T04:46:00.074168+0000","last_fullsized":"2024-03-26T04:46:00.074168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:39.810518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470809+0000","last_change":"2024-03-26T04:45:20.737703+0000","last_active":"2024-03-26T04:46:00.470809+0000","last_peered":"2024-03-26T04:46:00.470809+0000","last_clean":"2024-03-26T04:46:00.470809+0000","last_became_active":"2024-03-26T04:45:20.737397+0000","last_became_peered":"2024-03-26T04:45:20.737397+0000","last_unstale":"2024-03-26T04:46:00.470809+0000","last_undegraded":"2024-03-26T04:46:00.470809+0000","last_fullsized":"2024-03-26T04:46:00.470809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:39:52.818920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076803+0000","last_change":"2024-03-26T04:45:22.759227+0000","last_active":"2024-03-26T04:46:00.076803+0000","last_peered":"2024-03-26T04:46:00.076803+0000","last_clean":"2024-03-26T04:46:00.076803+0000","last_became_active":"2024-03-26T04:45:22.758820+0000","last_became_peered":"2024-03-26T04:45:22.758820+0000","last_unstale":"2024-03-26T04:46:00.076803+0000","last_undegraded":"2024-03-26T04:46:00.076803+0000","last_fullsized":"2024-03-26T04:46:00.076803+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:22:42.965112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073752+0000","last_change":"2024-03-26T04:45:26.781913+0000","last_active":"2024-03-26T04:46:00.073752+0000","last_peered":"2024-03-26T04:46:00.073752+0000","last_clean":"2024-03-26T04:46:00.073752+0000","last_became_active":"2024-03-26T04:45:26.781562+0000","last_became_peered":"2024-03-26T04:45:26.781562+0000","last_unstale":"2024-03-26T04:46:00.073752+0000","last_undegraded":"2024-03-26T04:46:00.073752+0000","last_fullsized":"2024-03-26T04:46:00.073752+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:17:52.863809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.200011+0000","last_change":"2024-03-26T04:45:24.778698+0000","last_active":"2024-03-26T04:46:00.200011+0000","last_peered":"2024-03-26T04:46:00.200011+0000","last_clean":"2024-03-26T04:46:00.200011+0000","last_became_active":"2024-03-26T04:45:24.778388+0000","last_became_peered":"2024-03-26T04:45:24.778388+0000","last_unstale":"2024-03-26T04:46:00.200011+0000","last_undegraded":"2024-03-26T04:46:00.200011+0000","last_fullsized":"2024-03-26T04:46:00.200011+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:19:09.788321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188373+0000","last_change":"2024-03-26T04:45:20.739547+0000","last_active":"2024-03-26T04:46:00.188373+0000","last_peered":"2024-03-26T04:46:00.188373+0000","last_clean":"2024-03-26T04:46:00.188373+0000","last_became_active":"2024-03-26T04:45:20.739256+0000","last_became_peered":"2024-03-26T04:45:20.739256+0000","last_unstale":"2024-03-26T04:46:00.188373+0000","last_undegraded":"2024-03-26T04:46:00.188373+0000","last_fullsized":"2024-03-26T04:46:00.188373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:34:23.837482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470708+0000","last_change":"2024-03-26T04:45:22.761083+0000","last_active":"2024-03-26T04:46:00.470708+0000","last_peered":"2024-03-26T04:46:00.470708+0000","last_clean":"2024-03-26T04:46:00.470708+0000","last_became_active":"2024-03-26T04:45:22.760776+0000","last_became_peered":"2024-03-26T04:45:22.760776+0000","last_unstale":"2024-03-26T04:46:00.470708+0000","last_undegraded":"2024-03-26T04:46:00.470708+0000","last_fullsized":"2024-03-26T04:46:00.470708+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:59:52.847474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470469+0000","last_change":"2024-03-26T04:45:26.788091+0000","last_active":"2024-03-26T04:46:00.470469+0000","last_peered":"2024-03-26T04:46:00.470469+0000","last_clean":"2024-03-26T04:46:00.470469+0000","last_became_active":"2024-03-26T04:45:26.787780+0000","last_became_peered":"2024-03-26T04:45:26.787780+0000","last_unstale":"2024-03-26T04:46:00.470469+0000","last_undegraded":"2024-03-26T04:46:00.470469+0000","last_fullsized":"2024-03-26T04:46:00.470469+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:26:45.315783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.807059+0000","last_change":"2024-03-26T04:45:24.779611+0000","last_active":"2024-03-26T04:46:35.807059+0000","last_peered":"2024-03-26T04:46:35.807059+0000","last_clean":"2024-03-26T04:46:35.807059+0000","last_became_active":"2024-03-26T04:45:24.778211+0000","last_became_peered":"2024-03-26T04:45:24.778211+0000","last_unstale":"2024-03-26T04:46:35.807059+0000","last_undegraded":"2024-03-26T04:46:35.807059+0000","last_fullsized":"2024-03-26T04:46:35.807059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:01:20.063792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469283+0000","last_change":"2024-03-26T04:45:20.741997+0000","last_active":"2024-03-26T04:46:00.469283+0000","last_peered":"2024-03-26T04:46:00.469283+0000","last_clean":"2024-03-26T04:46:00.469283+0000","last_became_active":"2024-03-26T04:45:20.741681+0000","last_became_peered":"2024-03-26T04:45:20.741681+0000","last_unstale":"2024-03-26T04:46:00.469283+0000","last_undegraded":"2024-03-26T04:46:00.469283+0000","last_fullsized":"2024-03-26T04:46:00.469283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:34:33.166949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199995+0000","last_change":"2024-03-26T04:45:22.750286+0000","last_active":"2024-03-26T04:46:00.199995+0000","last_peered":"2024-03-26T04:46:00.199995+0000","last_clean":"2024-03-26T04:46:00.199995+0000","last_became_active":"2024-03-26T04:45:22.749925+0000","last_became_peered":"2024-03-26T04:45:22.749925+0000","last_unstale":"2024-03-26T04:46:00.199995+0000","last_undegraded":"2024-03-26T04:46:00.199995+0000","last_fullsized":"2024-03-26T04:46:00.199995+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:06:20.335133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189228+0000","last_change":"2024-03-26T04:45:26.783513+0000","last_active":"2024-03-26T04:46:00.189228+0000","last_peered":"2024-03-26T04:46:00.189228+0000","last_clean":"2024-03-26T04:46:00.189228+0000","last_became_active":"2024-03-26T04:45:26.783159+0000","last_became_peered":"2024-03-26T04:45:26.783159+0000","last_unstale":"2024-03-26T04:46:00.189228+0000","last_undegraded":"2024-03-26T04:46:00.189228+0000","last_fullsized":"2024-03-26T04:46:00.189228+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:55.621387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471031+0000","last_change":"2024-03-26T04:45:24.781221+0000","last_active":"2024-03-26T04:46:00.471031+0000","last_peered":"2024-03-26T04:46:00.471031+0000","last_clean":"2024-03-26T04:46:00.471031+0000","last_became_active":"2024-03-26T04:45:24.779992+0000","last_became_peered":"2024-03-26T04:45:24.779992+0000","last_unstale":"2024-03-26T04:46:00.471031+0000","last_undegraded":"2024-03-26T04:46:00.471031+0000","last_fullsized":"2024-03-26T04:46:00.471031+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:53:59.760550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471048+0000","last_change":"2024-03-26T04:45:20.739468+0000","last_active":"2024-03-26T04:46:00.471048+0000","last_peered":"2024-03-26T04:46:00.471048+0000","last_clean":"2024-03-26T04:46:00.471048+0000","last_became_active":"2024-03-26T04:45:20.739162+0000","last_became_peered":"2024-03-26T04:45:20.739162+0000","last_unstale":"2024-03-26T04:46:00.471048+0000","last_undegraded":"2024-03-26T04:46:00.471048+0000","last_fullsized":"2024-03-26T04:46:00.471048+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:20.871210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188658+0000","last_change":"2024-03-26T04:45:22.758910+0000","last_active":"2024-03-26T04:46:00.188658+0000","last_peered":"2024-03-26T04:46:00.188658+0000","last_clean":"2024-03-26T04:46:00.188658+0000","last_became_active":"2024-03-26T04:45:22.758588+0000","last_became_peered":"2024-03-26T04:45:22.758588+0000","last_unstale":"2024-03-26T04:46:00.188658+0000","last_undegraded":"2024-03-26T04:46:00.188658+0000","last_fullsized":"2024-03-26T04:46:00.188658+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:28:00.884941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470309+0000","last_change":"2024-03-26T04:45:26.785967+0000","last_active":"2024-03-26T04:46:00.470309+0000","last_peered":"2024-03-26T04:46:00.470309+0000","last_clean":"2024-03-26T04:46:00.470309+0000","last_became_active":"2024-03-26T04:45:26.785682+0000","last_became_peered":"2024-03-26T04:45:26.785682+0000","last_unstale":"2024-03-26T04:46:00.470309+0000","last_undegraded":"2024-03-26T04:46:00.470309+0000","last_fullsized":"2024-03-26T04:46:00.470309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:09:32.286415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469540+0000","last_change":"2024-03-26T04:45:24.774208+0000","last_active":"2024-03-26T04:46:00.469540+0000","last_peered":"2024-03-26T04:46:00.469540+0000","last_clean":"2024-03-26T04:46:00.469540+0000","last_became_active":"2024-03-26T04:45:24.773883+0000","last_became_peered":"2024-03-26T04:45:24.773883+0000","last_unstale":"2024-03-26T04:46:00.469540+0000","last_undegraded":"2024-03-26T04:46:00.469540+0000","last_fullsized":"2024-03-26T04:46:00.469540+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:45.352012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471123+0000","last_change":"2024-03-26T04:45:20.741746+0000","last_active":"2024-03-26T04:46:00.471123+0000","last_peered":"2024-03-26T04:46:00.471123+0000","last_clean":"2024-03-26T04:46:00.471123+0000","last_became_active":"2024-03-26T04:45:20.741165+0000","last_became_peered":"2024-03-26T04:45:20.741165+0000","last_unstale":"2024-03-26T04:46:00.471123+0000","last_undegraded":"2024-03-26T04:46:00.471123+0000","last_fullsized":"2024-03-26T04:46:00.471123+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:19:04.836421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188288+0000","last_change":"2024-03-26T04:45:22.755755+0000","last_active":"2024-03-26T04:46:00.188288+0000","last_peered":"2024-03-26T04:46:00.188288+0000","last_clean":"2024-03-26T04:46:00.188288+0000","last_became_active":"2024-03-26T04:45:22.755440+0000","last_became_peered":"2024-03-26T04:45:22.755440+0000","last_unstale":"2024-03-26T04:46:00.188288+0000","last_undegraded":"2024-03-26T04:46:00.188288+0000","last_fullsized":"2024-03-26T04:46:00.188288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:58.905337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188403+0000","last_change":"2024-03-26T04:45:26.788244+0000","last_active":"2024-03-26T04:46:00.188403+0000","last_peered":"2024-03-26T04:46:00.188403+0000","last_clean":"2024-03-26T04:46:00.188403+0000","last_became_active":"2024-03-26T04:45:26.787918+0000","last_became_peered":"2024-03-26T04:45:26.787918+0000","last_unstale":"2024-03-26T04:46:00.188403+0000","last_undegraded":"2024-03-26T04:46:00.188403+0000","last_fullsized":"2024-03-26T04:46:00.188403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:56:19.821640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075303+0000","last_change":"2024-03-26T04:45:24.780301+0000","last_active":"2024-03-26T04:46:00.075303+0000","last_peered":"2024-03-26T04:46:00.075303+0000","last_clean":"2024-03-26T04:46:00.075303+0000","last_became_active":"2024-03-26T04:45:24.779817+0000","last_became_peered":"2024-03-26T04:45:24.779817+0000","last_unstale":"2024-03-26T04:46:00.075303+0000","last_undegraded":"2024-03-26T04:46:00.075303+0000","last_fullsized":"2024-03-26T04:46:00.075303+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:28:15.892797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073977+0000","last_change":"2024-03-26T04:45:20.749055+0000","last_active":"2024-03-26T04:46:00.073977+0000","last_peered":"2024-03-26T04:46:00.073977+0000","last_clean":"2024-03-26T04:46:00.073977+0000","last_became_active":"2024-03-26T04:45:20.748715+0000","last_became_peered":"2024-03-26T04:45:20.748715+0000","last_unstale":"2024-03-26T04:46:00.073977+0000","last_undegraded":"2024-03-26T04:46:00.073977+0000","last_fullsized":"2024-03-26T04:46:00.073977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:57:54.073215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075321+0000","last_change":"2024-03-26T04:45:22.760671+0000","last_active":"2024-03-26T04:46:00.075321+0000","last_peered":"2024-03-26T04:46:00.075321+0000","last_clean":"2024-03-26T04:46:00.075321+0000","last_became_active":"2024-03-26T04:45:22.760306+0000","last_became_peered":"2024-03-26T04:45:22.760306+0000","last_unstale":"2024-03-26T04:46:00.075321+0000","last_undegraded":"2024-03-26T04:46:00.075321+0000","last_fullsized":"2024-03-26T04:46:00.075321+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:23:10.513174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.198789+0000","last_change":"2024-03-26T04:45:26.790448+0000","last_active":"2024-03-26T04:46:00.198789+0000","last_peered":"2024-03-26T04:46:00.198789+0000","last_clean":"2024-03-26T04:46:00.198789+0000","last_became_active":"2024-03-26T04:45:26.790096+0000","last_became_peered":"2024-03-26T04:45:26.790096+0000","last_unstale":"2024-03-26T04:46:00.198789+0000","last_undegraded":"2024-03-26T04:46:00.198789+0000","last_fullsized":"2024-03-26T04:46:00.198789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:50:31.119087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189142+0000","last_change":"2024-03-26T04:45:24.780791+0000","last_active":"2024-03-26T04:46:00.189142+0000","last_peered":"2024-03-26T04:46:00.189142+0000","last_clean":"2024-03-26T04:46:00.189142+0000","last_became_active":"2024-03-26T04:45:24.780466+0000","last_became_peered":"2024-03-26T04:45:24.780466+0000","last_unstale":"2024-03-26T04:46:00.189142+0000","last_undegraded":"2024-03-26T04:46:00.189142+0000","last_fullsized":"2024-03-26T04:46:00.189142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:42:04.128173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471244+0000","last_change":"2024-03-26T04:45:20.740613+0000","last_active":"2024-03-26T04:46:00.471244+0000","last_peered":"2024-03-26T04:46:00.471244+0000","last_clean":"2024-03-26T04:46:00.471244+0000","last_became_active":"2024-03-26T04:45:20.740304+0000","last_became_peered":"2024-03-26T04:45:20.740304+0000","last_unstale":"2024-03-26T04:46:00.471244+0000","last_undegraded":"2024-03-26T04:46:00.471244+0000","last_fullsized":"2024-03-26T04:46:00.471244+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:05:04.102035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075139+0000","last_change":"2024-03-26T04:45:22.757790+0000","last_active":"2024-03-26T04:46:00.075139+0000","last_peered":"2024-03-26T04:46:00.075139+0000","last_clean":"2024-03-26T04:46:00.075139+0000","last_became_active":"2024-03-26T04:45:22.756742+0000","last_became_peered":"2024-03-26T04:45:22.756742+0000","last_unstale":"2024-03-26T04:46:00.075139+0000","last_undegraded":"2024-03-26T04:46:00.075139+0000","last_fullsized":"2024-03-26T04:46:00.075139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:12:27.104263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076003+0000","last_change":"2024-03-26T04:45:26.777995+0000","last_active":"2024-03-26T04:46:00.076003+0000","last_peered":"2024-03-26T04:46:00.076003+0000","last_clean":"2024-03-26T04:46:00.076003+0000","last_became_active":"2024-03-26T04:45:26.777394+0000","last_became_peered":"2024-03-26T04:45:26.777394+0000","last_unstale":"2024-03-26T04:46:00.076003+0000","last_undegraded":"2024-03-26T04:46:00.076003+0000","last_fullsized":"2024-03-26T04:46:00.076003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:19:53.611019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.806794+0000","last_change":"2024-03-26T04:45:24.781638+0000","last_active":"2024-03-26T04:46:35.806794+0000","last_peered":"2024-03-26T04:46:35.806794+0000","last_clean":"2024-03-26T04:46:35.806794+0000","last_became_active":"2024-03-26T04:45:24.781269+0000","last_became_peered":"2024-03-26T04:45:24.781269+0000","last_unstale":"2024-03-26T04:46:35.806794+0000","last_undegraded":"2024-03-26T04:46:35.806794+0000","last_fullsized":"2024-03-26T04:46:35.806794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:59:41.625581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199497+0000","last_change":"2024-03-26T04:45:20.744611+0000","last_active":"2024-03-26T04:46:00.199497+0000","last_peered":"2024-03-26T04:46:00.199497+0000","last_clean":"2024-03-26T04:46:00.199497+0000","last_became_active":"2024-03-26T04:45:20.744113+0000","last_became_peered":"2024-03-26T04:45:20.744113+0000","last_unstale":"2024-03-26T04:46:00.199497+0000","last_undegraded":"2024-03-26T04:46:00.199497+0000","last_fullsized":"2024-03-26T04:46:00.199497+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:59.753246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471352+0000","last_change":"2024-03-26T04:45:22.753202+0000","last_active":"2024-03-26T04:46:00.471352+0000","last_peered":"2024-03-26T04:46:00.471352+0000","last_clean":"2024-03-26T04:46:00.471352+0000","last_became_active":"2024-03-26T04:45:22.752239+0000","last_became_peered":"2024-03-26T04:45:22.752239+0000","last_unstale":"2024-03-26T04:46:00.471352+0000","last_undegraded":"2024-03-26T04:46:00.471352+0000","last_fullsized":"2024-03-26T04:46:00.471352+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:40:52.537201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199732+0000","last_change":"2024-03-26T04:45:26.789517+0000","last_active":"2024-03-26T04:46:00.199732+0000","last_peered":"2024-03-26T04:46:00.199732+0000","last_clean":"2024-03-26T04:46:00.199732+0000","last_became_active":"2024-03-26T04:45:26.789195+0000","last_became_peered":"2024-03-26T04:45:26.789195+0000","last_unstale":"2024-03-26T04:46:00.199732+0000","last_undegraded":"2024-03-26T04:46:00.199732+0000","last_fullsized":"2024-03-26T04:46:00.199732+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:11:12.114878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.807077+0000","last_change":"2024-03-26T04:45:24.784181+0000","last_active":"2024-03-26T04:46:35.807077+0000","last_peered":"2024-03-26T04:46:35.807077+0000","last_clean":"2024-03-26T04:46:35.807077+0000","last_became_active":"2024-03-26T04:45:24.783869+0000","last_became_peered":"2024-03-26T04:45:24.783869+0000","last_unstale":"2024-03-26T04:46:35.807077+0000","last_undegraded":"2024-03-26T04:46:35.807077+0000","last_fullsized":"2024-03-26T04:46:35.807077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:43:28.834081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469850+0000","last_change":"2024-03-26T04:45:20.740730+0000","last_active":"2024-03-26T04:46:00.469850+0000","last_peered":"2024-03-26T04:46:00.469850+0000","last_clean":"2024-03-26T04:46:00.469850+0000","last_became_active":"2024-03-26T04:45:20.739873+0000","last_became_peered":"2024-03-26T04:45:20.739873+0000","last_unstale":"2024-03-26T04:46:00.469850+0000","last_undegraded":"2024-03-26T04:46:00.469850+0000","last_fullsized":"2024-03-26T04:46:00.469850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:48:55.926794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189400+0000","last_change":"2024-03-26T04:45:22.761397+0000","last_active":"2024-03-26T04:46:00.189400+0000","last_peered":"2024-03-26T04:46:00.189400+0000","last_clean":"2024-03-26T04:46:00.189400+0000","last_became_active":"2024-03-26T04:45:22.760937+0000","last_became_peered":"2024-03-26T04:45:22.760937+0000","last_unstale":"2024-03-26T04:46:00.189400+0000","last_undegraded":"2024-03-26T04:46:00.189400+0000","last_fullsized":"2024-03-26T04:46:00.189400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:40:01.328517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469911+0000","last_change":"2024-03-26T04:45:26.782497+0000","last_active":"2024-03-26T04:46:00.469911+0000","last_peered":"2024-03-26T04:46:00.469911+0000","last_clean":"2024-03-26T04:46:00.469911+0000","last_became_active":"2024-03-26T04:45:26.782215+0000","last_became_peered":"2024-03-26T04:45:26.782215+0000","last_unstale":"2024-03-26T04:46:00.469911+0000","last_undegraded":"2024-03-26T04:46:00.469911+0000","last_fullsized":"2024-03-26T04:46:00.469911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:13.088620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469833+0000","last_change":"2024-03-26T04:45:24.784123+0000","last_active":"2024-03-26T04:46:00.469833+0000","last_peered":"2024-03-26T04:46:00.469833+0000","last_clean":"2024-03-26T04:46:00.469833+0000","last_became_active":"2024-03-26T04:45:24.783786+0000","last_became_peered":"2024-03-26T04:45:24.783786+0000","last_unstale":"2024-03-26T04:46:00.469833+0000","last_undegraded":"2024-03-26T04:46:00.469833+0000","last_fullsized":"2024-03-26T04:46:00.469833+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:52:14.978506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469648+0000","last_change":"2024-03-26T04:45:20.736887+0000","last_active":"2024-03-26T04:46:00.469648+0000","last_peered":"2024-03-26T04:46:00.469648+0000","last_clean":"2024-03-26T04:46:00.469648+0000","last_became_active":"2024-03-26T04:45:20.736259+0000","last_became_peered":"2024-03-26T04:45:20.736259+0000","last_unstale":"2024-03-26T04:46:00.469648+0000","last_undegraded":"2024-03-26T04:46:00.469648+0000","last_fullsized":"2024-03-26T04:46:00.469648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:10:39.930806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189850+0000","last_change":"2024-03-26T04:45:22.757325+0000","last_active":"2024-03-26T04:46:00.189850+0000","last_peered":"2024-03-26T04:46:00.189850+0000","last_clean":"2024-03-26T04:46:00.189850+0000","last_became_active":"2024-03-26T04:45:22.756989+0000","last_became_peered":"2024-03-26T04:45:22.756989+0000","last_unstale":"2024-03-26T04:46:00.189850+0000","last_undegraded":"2024-03-26T04:46:00.189850+0000","last_fullsized":"2024-03-26T04:46:00.189850+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:48:32.854181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076259+0000","last_change":"2024-03-26T04:45:26.780364+0000","last_active":"2024-03-26T04:46:00.076259+0000","last_peered":"2024-03-26T04:46:00.076259+0000","last_clean":"2024-03-26T04:46:00.076259+0000","last_became_active":"2024-03-26T04:45:26.780016+0000","last_became_peered":"2024-03-26T04:45:26.780016+0000","last_unstale":"2024-03-26T04:46:00.076259+0000","last_undegraded":"2024-03-26T04:46:00.076259+0000","last_fullsized":"2024-03-26T04:46:00.076259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:04:24.907474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189970+0000","last_change":"2024-03-26T04:45:24.775319+0000","last_active":"2024-03-26T04:46:00.189970+0000","last_peered":"2024-03-26T04:46:00.189970+0000","last_clean":"2024-03-26T04:46:00.189970+0000","last_became_active":"2024-03-26T04:45:24.775001+0000","last_became_peered":"2024-03-26T04:45:24.775001+0000","last_unstale":"2024-03-26T04:46:00.189970+0000","last_undegraded":"2024-03-26T04:46:00.189970+0000","last_fullsized":"2024-03-26T04:46:00.189970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:16:27.717843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075909+0000","last_change":"2024-03-26T04:45:20.746067+0000","last_active":"2024-03-26T04:46:00.075909+0000","last_peered":"2024-03-26T04:46:00.075909+0000","last_clean":"2024-03-26T04:46:00.075909+0000","last_became_active":"2024-03-26T04:45:20.745745+0000","last_became_peered":"2024-03-26T04:45:20.745745+0000","last_unstale":"2024-03-26T04:46:00.075909+0000","last_undegraded":"2024-03-26T04:46:00.075909+0000","last_fullsized":"2024-03-26T04:46:00.075909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:51:49.254140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199859+0000","last_change":"2024-03-26T04:45:22.750074+0000","last_active":"2024-03-26T04:46:00.199859+0000","last_peered":"2024-03-26T04:46:00.199859+0000","last_clean":"2024-03-26T04:46:00.199859+0000","last_became_active":"2024-03-26T04:45:22.749633+0000","last_became_peered":"2024-03-26T04:45:22.749633+0000","last_unstale":"2024-03-26T04:46:00.199859+0000","last_undegraded":"2024-03-26T04:46:00.199859+0000","last_fullsized":"2024-03-26T04:46:00.199859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:30:18.952438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189188+0000","last_change":"2024-03-26T04:45:26.783345+0000","last_active":"2024-03-26T04:46:00.189188+0000","last_peered":"2024-03-26T04:46:00.189188+0000","last_clean":"2024-03-26T04:46:00.189188+0000","last_became_active":"2024-03-26T04:45:26.782972+0000","last_became_peered":"2024-03-26T04:45:26.782972+0000","last_unstale":"2024-03-26T04:46:00.189188+0000","last_undegraded":"2024-03-26T04:46:00.189188+0000","last_fullsized":"2024-03-26T04:46:00.189188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:21:49.919377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076692+0000","last_change":"2024-03-26T04:45:20.744069+0000","last_active":"2024-03-26T04:46:00.076692+0000","last_peered":"2024-03-26T04:46:00.076692+0000","last_clean":"2024-03-26T04:46:00.076692+0000","last_became_active":"2024-03-26T04:45:20.743767+0000","last_became_peered":"2024-03-26T04:45:20.743767+0000","last_unstale":"2024-03-26T04:46:00.076692+0000","last_undegraded":"2024-03-26T04:46:00.076692+0000","last_fullsized":"2024-03-26T04:46:00.076692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:15.121354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075184+0000","last_change":"2024-03-26T04:45:22.756536+0000","last_active":"2024-03-26T04:46:00.075184+0000","last_peered":"2024-03-26T04:46:00.075184+0000","last_clean":"2024-03-26T04:46:00.075184+0000","last_became_active":"2024-03-26T04:45:22.756239+0000","last_became_peered":"2024-03-26T04:45:22.756239+0000","last_unstale":"2024-03-26T04:46:00.075184+0000","last_undegraded":"2024-03-26T04:46:00.075184+0000","last_fullsized":"2024-03-26T04:46:00.075184+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:27:46.056197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075820+0000","last_change":"2024-03-26T04:45:24.772349+0000","last_active":"2024-03-26T04:46:00.075820+0000","last_peered":"2024-03-26T04:46:00.075820+0000","last_clean":"2024-03-26T04:46:00.075820+0000","last_became_active":"2024-03-26T04:45:24.771977+0000","last_became_peered":"2024-03-26T04:45:24.771977+0000","last_unstale":"2024-03-26T04:46:00.075820+0000","last_undegraded":"2024-03-26T04:46:00.075820+0000","last_fullsized":"2024-03-26T04:46:00.075820+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:22:53.791394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.198801+0000","last_change":"2024-03-26T04:45:26.784601+0000","last_active":"2024-03-26T04:46:00.198801+0000","last_peered":"2024-03-26T04:46:00.198801+0000","last_clean":"2024-03-26T04:46:00.198801+0000","last_became_active":"2024-03-26T04:45:26.784262+0000","last_became_peered":"2024-03-26T04:45:26.784262+0000","last_unstale":"2024-03-26T04:46:00.198801+0000","last_undegraded":"2024-03-26T04:46:00.198801+0000","last_fullsized":"2024-03-26T04:46:00.198801+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:32.073700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073918+0000","last_change":"2024-03-26T04:45:20.748989+0000","last_active":"2024-03-26T04:46:00.073918+0000","last_peered":"2024-03-26T04:46:00.073918+0000","last_clean":"2024-03-26T04:46:00.073918+0000","last_became_active":"2024-03-26T04:45:20.748624+0000","last_became_peered":"2024-03-26T04:45:20.748624+0000","last_unstale":"2024-03-26T04:46:00.073918+0000","last_undegraded":"2024-03-26T04:46:00.073918+0000","last_fullsized":"2024-03-26T04:46:00.073918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:23:13.070723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075267+0000","last_change":"2024-03-26T04:45:22.756873+0000","last_active":"2024-03-26T04:46:00.075267+0000","last_peered":"2024-03-26T04:46:00.075267+0000","last_clean":"2024-03-26T04:46:00.075267+0000","last_became_active":"2024-03-26T04:45:22.756184+0000","last_became_peered":"2024-03-26T04:45:22.756184+0000","last_unstale":"2024-03-26T04:46:00.075267+0000","last_undegraded":"2024-03-26T04:46:00.075267+0000","last_fullsized":"2024-03-26T04:46:00.075267+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:22:38.661330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.806736+0000","last_change":"2024-03-26T04:45:24.776787+0000","last_active":"2024-03-26T04:46:35.806736+0000","last_peered":"2024-03-26T04:46:35.806736+0000","last_clean":"2024-03-26T04:46:35.806736+0000","last_became_active":"2024-03-26T04:45:24.776172+0000","last_became_peered":"2024-03-26T04:45:24.776172+0000","last_unstale":"2024-03-26T04:46:35.806736+0000","last_undegraded":"2024-03-26T04:46:35.806736+0000","last_fullsized":"2024-03-26T04:46:35.806736+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:40:30.435694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.189810+0000","last_change":"2024-03-26T04:45:26.781682+0000","last_active":"2024-03-26T04:46:00.189810+0000","last_peered":"2024-03-26T04:46:00.189810+0000","last_clean":"2024-03-26T04:46:00.189810+0000","last_became_active":"2024-03-26T04:45:26.781280+0000","last_became_peered":"2024-03-26T04:45:26.781280+0000","last_unstale":"2024-03-26T04:46:00.189810+0000","last_undegraded":"2024-03-26T04:46:00.189810+0000","last_fullsized":"2024-03-26T04:46:00.189810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:20:27.101384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469946+0000","last_change":"2024-03-26T04:45:20.742917+0000","last_active":"2024-03-26T04:46:00.469946+0000","last_peered":"2024-03-26T04:46:00.469946+0000","last_clean":"2024-03-26T04:46:00.469946+0000","last_became_active":"2024-03-26T04:45:20.742643+0000","last_became_peered":"2024-03-26T04:45:20.742643+0000","last_unstale":"2024-03-26T04:46:00.469946+0000","last_undegraded":"2024-03-26T04:46:00.469946+0000","last_fullsized":"2024-03-26T04:46:00.469946+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:33:55.822942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"67'34","reported_seq":106,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.806784+0000","last_change":"2024-03-26T04:45:22.761391+0000","last_active":"2024-03-26T04:46:35.806784+0000","last_peered":"2024-03-26T04:46:35.806784+0000","last_clean":"2024-03-26T04:46:35.806784+0000","last_became_active":"2024-03-26T04:45:22.761021+0000","last_became_peered":"2024-03-26T04:45:22.761021+0000","last_unstale":"2024-03-26T04:46:35.806784+0000","last_undegraded":"2024-03-26T04:46:35.806784+0000","last_fullsized":"2024-03-26T04:46:35.806784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:14:47.623733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199993+0000","last_change":"2024-03-26T04:45:24.777638+0000","last_active":"2024-03-26T04:46:00.199993+0000","last_peered":"2024-03-26T04:46:00.199993+0000","last_clean":"2024-03-26T04:46:00.199993+0000","last_became_active":"2024-03-26T04:45:24.777326+0000","last_became_peered":"2024-03-26T04:45:24.777326+0000","last_unstale":"2024-03-26T04:46:00.199993+0000","last_undegraded":"2024-03-26T04:46:00.199993+0000","last_fullsized":"2024-03-26T04:46:00.199993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:05:57.136042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471480+0000","last_change":"2024-03-26T04:45:26.785143+0000","last_active":"2024-03-26T04:46:00.471480+0000","last_peered":"2024-03-26T04:46:00.471480+0000","last_clean":"2024-03-26T04:46:00.471480+0000","last_became_active":"2024-03-26T04:45:26.784843+0000","last_became_peered":"2024-03-26T04:45:26.784843+0000","last_unstale":"2024-03-26T04:46:00.471480+0000","last_undegraded":"2024-03-26T04:46:00.471480+0000","last_fullsized":"2024-03-26T04:46:00.471480+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:37:00.953324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"68'5","reported_seq":118,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:39.934509+0000","last_change":"2024-03-26T04:45:24.765271+0000","last_active":"2024-03-26T04:46:39.934509+0000","last_peered":"2024-03-26T04:46:39.934509+0000","last_clean":"2024-03-26T04:46:39.934509+0000","last_became_active":"2024-03-26T04:45:20.735553+0000","last_became_peered":"2024-03-26T04:45:20.735553+0000","last_unstale":"2024-03-26T04:46:39.934509+0000","last_undegraded":"2024-03-26T04:46:39.934509+0000","last_fullsized":"2024-03-26T04:46:39.934509+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:02:42.388391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030280620000000002,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075979+0000","last_change":"2024-03-26T04:45:20.740834+0000","last_active":"2024-03-26T04:46:00.075979+0000","last_peered":"2024-03-26T04:46:00.075979+0000","last_clean":"2024-03-26T04:46:00.075979+0000","last_became_active":"2024-03-26T04:45:20.740528+0000","last_became_peered":"2024-03-26T04:45:20.740528+0000","last_unstale":"2024-03-26T04:46:00.075979+0000","last_undegraded":"2024-03-26T04:46:00.075979+0000","last_fullsized":"2024-03-26T04:46:00.075979+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:35:07.801959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":27,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076626+0000","last_change":"2024-03-26T04:45:45.921771+0000","last_active":"2024-03-26T04:46:00.076626+0000","last_peered":"2024-03-26T04:46:00.076626+0000","last_clean":"2024-03-26T04:46:00.076626+0000","last_became_active":"2024-03-26T04:45:45.906012+0000","last_became_peered":"2024-03-26T04:45:45.906012+0000","last_unstale":"2024-03-26T04:46:00.076626+0000","last_undegraded":"2024-03-26T04:46:00.076626+0000","last_fullsized":"2024-03-26T04:46:00.076626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:14:59.192089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076569+0000","last_change":"2024-03-26T04:45:24.779099+0000","last_active":"2024-03-26T04:46:00.076569+0000","last_peered":"2024-03-26T04:46:00.076569+0000","last_clean":"2024-03-26T04:46:00.076569+0000","last_became_active":"2024-03-26T04:45:24.778722+0000","last_became_peered":"2024-03-26T04:45:24.778722+0000","last_unstale":"2024-03-26T04:46:00.076569+0000","last_undegraded":"2024-03-26T04:46:00.076569+0000","last_fullsized":"2024-03-26T04:46:00.076569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:47.337053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470905+0000","last_change":"2024-03-26T04:45:26.786252+0000","last_active":"2024-03-26T04:46:00.470905+0000","last_peered":"2024-03-26T04:46:00.470905+0000","last_clean":"2024-03-26T04:46:00.470905+0000","last_became_active":"2024-03-26T04:45:26.785929+0000","last_became_peered":"2024-03-26T04:45:26.785929+0000","last_unstale":"2024-03-26T04:46:00.470905+0000","last_undegraded":"2024-03-26T04:46:00.470905+0000","last_fullsized":"2024-03-26T04:46:00.470905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:36:23.721970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188410+0000","last_change":"2024-03-26T04:45:24.771645+0000","last_active":"2024-03-26T04:46:00.188410+0000","last_peered":"2024-03-26T04:46:00.188410+0000","last_clean":"2024-03-26T04:46:00.188410+0000","last_became_active":"2024-03-26T04:45:20.738369+0000","last_became_peered":"2024-03-26T04:45:20.738369+0000","last_unstale":"2024-03-26T04:46:00.188410+0000","last_undegraded":"2024-03-26T04:46:00.188410+0000","last_fullsized":"2024-03-26T04:46:00.188410+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:19:57.782576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020341589999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469742+0000","last_change":"2024-03-26T04:45:20.743758+0000","last_active":"2024-03-26T04:46:00.469742+0000","last_peered":"2024-03-26T04:46:00.469742+0000","last_clean":"2024-03-26T04:46:00.469742+0000","last_became_active":"2024-03-26T04:45:20.743351+0000","last_became_peered":"2024-03-26T04:45:20.743351+0000","last_unstale":"2024-03-26T04:46:00.469742+0000","last_undegraded":"2024-03-26T04:46:00.469742+0000","last_fullsized":"2024-03-26T04:46:00.469742+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:41.152873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'128","reported_seq":275,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:02.181048+0000","last_change":"2024-03-26T04:45:10.297709+0000","last_active":"2024-03-26T04:46:02.181048+0000","last_peered":"2024-03-26T04:46:02.181048+0000","last_clean":"2024-03-26T04:46:02.181048+0000","last_became_active":"2024-03-26T04:45:10.297289+0000","last_became_peered":"2024-03-26T04:45:10.297289+0000","last_unstale":"2024-03-26T04:46:02.181048+0000","last_undegraded":"2024-03-26T04:46:02.181048+0000","last_fullsized":"2024-03-26T04:46:02.181048+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:43:00.973945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:43:00.973945+0000","last_clean_scrub_stamp":"2024-03-26T04:43:00.973945+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:59:45.901811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.807103+0000","last_change":"2024-03-26T04:45:22.755358+0000","last_active":"2024-03-26T04:46:35.807103+0000","last_peered":"2024-03-26T04:46:35.807103+0000","last_clean":"2024-03-26T04:46:35.807103+0000","last_became_active":"2024-03-26T04:45:22.754994+0000","last_became_peered":"2024-03-26T04:45:22.754994+0000","last_unstale":"2024-03-26T04:46:35.807103+0000","last_undegraded":"2024-03-26T04:46:35.807103+0000","last_fullsized":"2024-03-26T04:46:35.807103+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:12:22.098046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470857+0000","last_change":"2024-03-26T04:45:24.779972+0000","last_active":"2024-03-26T04:46:00.470857+0000","last_peered":"2024-03-26T04:46:00.470857+0000","last_clean":"2024-03-26T04:46:00.470857+0000","last_became_active":"2024-03-26T04:45:24.779640+0000","last_became_peered":"2024-03-26T04:45:24.779640+0000","last_unstale":"2024-03-26T04:46:00.470857+0000","last_undegraded":"2024-03-26T04:46:00.470857+0000","last_fullsized":"2024-03-26T04:46:00.470857+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:52:16.161546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073679+0000","last_change":"2024-03-26T04:45:26.786569+0000","last_active":"2024-03-26T04:46:00.073679+0000","last_peered":"2024-03-26T04:46:00.073679+0000","last_clean":"2024-03-26T04:46:00.073679+0000","last_became_active":"2024-03-26T04:45:26.786242+0000","last_became_peered":"2024-03-26T04:45:26.786242+0000","last_unstale":"2024-03-26T04:46:00.073679+0000","last_undegraded":"2024-03-26T04:46:00.073679+0000","last_fullsized":"2024-03-26T04:46:00.073679+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:45:13.961941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199169+0000","last_change":"2024-03-26T04:45:24.770350+0000","last_active":"2024-03-26T04:46:00.199169+0000","last_peered":"2024-03-26T04:46:00.199169+0000","last_clean":"2024-03-26T04:46:00.199169+0000","last_became_active":"2024-03-26T04:45:20.745164+0000","last_became_peered":"2024-03-26T04:45:20.745164+0000","last_unstale":"2024-03-26T04:46:00.199169+0000","last_undegraded":"2024-03-26T04:46:00.199169+0000","last_fullsized":"2024-03-26T04:46:00.199169+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:19:45.864436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00118863,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469274+0000","last_change":"2024-03-26T04:45:20.749345+0000","last_active":"2024-03-26T04:46:00.469274+0000","last_peered":"2024-03-26T04:46:00.469274+0000","last_clean":"2024-03-26T04:46:00.469274+0000","last_became_active":"2024-03-26T04:45:20.749063+0000","last_became_peered":"2024-03-26T04:45:20.749063+0000","last_unstale":"2024-03-26T04:46:00.469274+0000","last_undegraded":"2024-03-26T04:46:00.469274+0000","last_fullsized":"2024-03-26T04:46:00.469274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:01:32.095852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199744+0000","last_change":"2024-03-26T04:45:22.754658+0000","last_active":"2024-03-26T04:46:00.199744+0000","last_peered":"2024-03-26T04:46:00.199744+0000","last_clean":"2024-03-26T04:46:00.199744+0000","last_became_active":"2024-03-26T04:45:22.754226+0000","last_became_peered":"2024-03-26T04:45:22.754226+0000","last_unstale":"2024-03-26T04:46:00.199744+0000","last_undegraded":"2024-03-26T04:46:00.199744+0000","last_fullsized":"2024-03-26T04:46:00.199744+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:26:17.360139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470819+0000","last_change":"2024-03-26T04:45:24.783225+0000","last_active":"2024-03-26T04:46:00.470819+0000","last_peered":"2024-03-26T04:46:00.470819+0000","last_clean":"2024-03-26T04:46:00.470819+0000","last_became_active":"2024-03-26T04:45:24.782885+0000","last_became_peered":"2024-03-26T04:45:24.782885+0000","last_unstale":"2024-03-26T04:46:00.470819+0000","last_undegraded":"2024-03-26T04:46:00.470819+0000","last_fullsized":"2024-03-26T04:46:00.470819+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:49:49.195040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073133+0000","last_change":"2024-03-26T04:45:26.784002+0000","last_active":"2024-03-26T04:46:00.073133+0000","last_peered":"2024-03-26T04:46:00.073133+0000","last_clean":"2024-03-26T04:46:00.073133+0000","last_became_active":"2024-03-26T04:45:26.783676+0000","last_became_peered":"2024-03-26T04:45:26.783676+0000","last_unstale":"2024-03-26T04:46:00.073133+0000","last_undegraded":"2024-03-26T04:46:00.073133+0000","last_fullsized":"2024-03-26T04:46:00.073133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:19.711511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076344+0000","last_change":"2024-03-26T04:45:45.906497+0000","last_active":"2024-03-26T04:46:00.076344+0000","last_peered":"2024-03-26T04:46:00.076344+0000","last_clean":"2024-03-26T04:46:00.076344+0000","last_became_active":"2024-03-26T04:45:45.906096+0000","last_became_peered":"2024-03-26T04:45:45.906096+0000","last_unstale":"2024-03-26T04:46:00.076344+0000","last_undegraded":"2024-03-26T04:46:00.076344+0000","last_fullsized":"2024-03-26T04:46:00.076344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:21:38.012449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199206+0000","last_change":"2024-03-26T04:45:22.749002+0000","last_active":"2024-03-26T04:46:00.199206+0000","last_peered":"2024-03-26T04:46:00.199206+0000","last_clean":"2024-03-26T04:46:00.199206+0000","last_became_active":"2024-03-26T04:45:22.748634+0000","last_became_peered":"2024-03-26T04:45:22.748634+0000","last_unstale":"2024-03-26T04:46:00.199206+0000","last_undegraded":"2024-03-26T04:46:00.199206+0000","last_fullsized":"2024-03-26T04:46:00.199206+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:08:27.067377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470905+0000","last_change":"2024-03-26T04:45:24.775915+0000","last_active":"2024-03-26T04:46:00.470905+0000","last_peered":"2024-03-26T04:46:00.470905+0000","last_clean":"2024-03-26T04:46:00.470905+0000","last_became_active":"2024-03-26T04:45:24.775605+0000","last_became_peered":"2024-03-26T04:45:24.775605+0000","last_unstale":"2024-03-26T04:46:00.470905+0000","last_undegraded":"2024-03-26T04:46:00.470905+0000","last_fullsized":"2024-03-26T04:46:00.470905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:02:03.768569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075606+0000","last_change":"2024-03-26T04:45:26.781707+0000","last_active":"2024-03-26T04:46:00.075606+0000","last_peered":"2024-03-26T04:46:00.075606+0000","last_clean":"2024-03-26T04:46:00.075606+0000","last_became_active":"2024-03-26T04:45:26.781390+0000","last_became_peered":"2024-03-26T04:45:26.781390+0000","last_unstale":"2024-03-26T04:46:00.075606+0000","last_undegraded":"2024-03-26T04:46:00.075606+0000","last_fullsized":"2024-03-26T04:46:00.075606+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:37.471944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075624+0000","last_change":"2024-03-26T04:45:20.741148+0000","last_active":"2024-03-26T04:46:00.075624+0000","last_peered":"2024-03-26T04:46:00.075624+0000","last_clean":"2024-03-26T04:46:00.075624+0000","last_became_active":"2024-03-26T04:45:20.740818+0000","last_became_peered":"2024-03-26T04:45:20.740818+0000","last_unstale":"2024-03-26T04:46:00.075624+0000","last_undegraded":"2024-03-26T04:46:00.075624+0000","last_fullsized":"2024-03-26T04:46:00.075624+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:51.332314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"67'30","reported_seq":85,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188166+0000","last_change":"2024-03-26T04:45:22.754301+0000","last_active":"2024-03-26T04:46:00.188166+0000","last_peered":"2024-03-26T04:46:00.188166+0000","last_clean":"2024-03-26T04:46:00.188166+0000","last_became_active":"2024-03-26T04:45:22.753895+0000","last_became_peered":"2024-03-26T04:45:22.753895+0000","last_unstale":"2024-03-26T04:46:00.188166+0000","last_undegraded":"2024-03-26T04:46:00.188166+0000","last_fullsized":"2024-03-26T04:46:00.188166+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:05:51.617244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075299+0000","last_change":"2024-03-26T04:45:24.774608+0000","last_active":"2024-03-26T04:46:00.075299+0000","last_peered":"2024-03-26T04:46:00.075299+0000","last_clean":"2024-03-26T04:46:00.075299+0000","last_became_active":"2024-03-26T04:45:24.773823+0000","last_became_peered":"2024-03-26T04:45:24.773823+0000","last_unstale":"2024-03-26T04:46:00.075299+0000","last_undegraded":"2024-03-26T04:46:00.075299+0000","last_fullsized":"2024-03-26T04:46:00.075299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:15:35.588693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470355+0000","last_change":"2024-03-26T04:45:26.786034+0000","last_active":"2024-03-26T04:46:00.470355+0000","last_peered":"2024-03-26T04:46:00.470355+0000","last_clean":"2024-03-26T04:46:00.470355+0000","last_became_active":"2024-03-26T04:45:26.785734+0000","last_became_peered":"2024-03-26T04:45:26.785734+0000","last_unstale":"2024-03-26T04:46:00.470355+0000","last_undegraded":"2024-03-26T04:46:00.470355+0000","last_fullsized":"2024-03-26T04:46:00.470355+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:51:38.718687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470957+0000","last_change":"2024-03-26T04:45:20.738453+0000","last_active":"2024-03-26T04:46:00.470957+0000","last_peered":"2024-03-26T04:46:00.470957+0000","last_clean":"2024-03-26T04:46:00.470957+0000","last_became_active":"2024-03-26T04:45:20.738132+0000","last_became_peered":"2024-03-26T04:45:20.738132+0000","last_unstale":"2024-03-26T04:46:00.470957+0000","last_undegraded":"2024-03-26T04:46:00.470957+0000","last_fullsized":"2024-03-26T04:46:00.470957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:20.826382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075156+0000","last_change":"2024-03-26T04:45:22.754771+0000","last_active":"2024-03-26T04:46:00.075156+0000","last_peered":"2024-03-26T04:46:00.075156+0000","last_clean":"2024-03-26T04:46:00.075156+0000","last_became_active":"2024-03-26T04:45:22.754436+0000","last_became_peered":"2024-03-26T04:45:22.754436+0000","last_unstale":"2024-03-26T04:46:00.075156+0000","last_undegraded":"2024-03-26T04:46:00.075156+0000","last_fullsized":"2024-03-26T04:46:00.075156+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:56.003046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.806700+0000","last_change":"2024-03-26T04:45:24.778428+0000","last_active":"2024-03-26T04:46:35.806700+0000","last_peered":"2024-03-26T04:46:35.806700+0000","last_clean":"2024-03-26T04:46:35.806700+0000","last_became_active":"2024-03-26T04:45:24.778015+0000","last_became_peered":"2024-03-26T04:45:24.778015+0000","last_unstale":"2024-03-26T04:46:35.806700+0000","last_undegraded":"2024-03-26T04:46:35.806700+0000","last_fullsized":"2024-03-26T04:46:35.806700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:39.901802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.188946+0000","last_change":"2024-03-26T04:45:26.786936+0000","last_active":"2024-03-26T04:46:00.188946+0000","last_peered":"2024-03-26T04:46:00.188946+0000","last_clean":"2024-03-26T04:46:00.188946+0000","last_became_active":"2024-03-26T04:45:26.786635+0000","last_became_peered":"2024-03-26T04:45:26.786635+0000","last_unstale":"2024-03-26T04:46:00.188946+0000","last_undegraded":"2024-03-26T04:46:00.188946+0000","last_fullsized":"2024-03-26T04:46:00.188946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:30.058065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470665+0000","last_change":"2024-03-26T04:45:24.778189+0000","last_active":"2024-03-26T04:46:00.470665+0000","last_peered":"2024-03-26T04:46:00.470665+0000","last_clean":"2024-03-26T04:46:00.470665+0000","last_became_active":"2024-03-26T04:45:24.777853+0000","last_became_peered":"2024-03-26T04:45:24.777853+0000","last_unstale":"2024-03-26T04:46:00.470665+0000","last_undegraded":"2024-03-26T04:46:00.470665+0000","last_fullsized":"2024-03-26T04:46:00.470665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:07:50.814289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469477+0000","last_change":"2024-03-26T04:45:20.749005+0000","last_active":"2024-03-26T04:46:00.469477+0000","last_peered":"2024-03-26T04:46:00.469477+0000","last_clean":"2024-03-26T04:46:00.469477+0000","last_became_active":"2024-03-26T04:45:20.748662+0000","last_became_peered":"2024-03-26T04:45:20.748662+0000","last_unstale":"2024-03-26T04:46:00.469477+0000","last_undegraded":"2024-03-26T04:46:00.469477+0000","last_fullsized":"2024-03-26T04:46:00.469477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:12:17.013354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076189+0000","last_change":"2024-03-26T04:45:22.756409+0000","last_active":"2024-03-26T04:46:00.076189+0000","last_peered":"2024-03-26T04:46:00.076189+0000","last_clean":"2024-03-26T04:46:00.076189+0000","last_became_active":"2024-03-26T04:45:22.756082+0000","last_became_peered":"2024-03-26T04:45:22.756082+0000","last_unstale":"2024-03-26T04:46:00.076189+0000","last_undegraded":"2024-03-26T04:46:00.076189+0000","last_fullsized":"2024-03-26T04:46:00.076189+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:38:09.542742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471573+0000","last_change":"2024-03-26T04:45:26.790427+0000","last_active":"2024-03-26T04:46:00.471573+0000","last_peered":"2024-03-26T04:46:00.471573+0000","last_clean":"2024-03-26T04:46:00.471573+0000","last_became_active":"2024-03-26T04:45:26.789473+0000","last_became_peered":"2024-03-26T04:45:26.789473+0000","last_unstale":"2024-03-26T04:46:00.471573+0000","last_undegraded":"2024-03-26T04:46:00.471573+0000","last_fullsized":"2024-03-26T04:46:00.471573+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:39.960035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.807137+0000","last_change":"2024-03-26T04:45:24.776004+0000","last_active":"2024-03-26T04:46:35.807137+0000","last_peered":"2024-03-26T04:46:35.807137+0000","last_clean":"2024-03-26T04:46:35.807137+0000","last_became_active":"2024-03-26T04:45:24.775501+0000","last_became_peered":"2024-03-26T04:45:24.775501+0000","last_unstale":"2024-03-26T04:46:35.807137+0000","last_undegraded":"2024-03-26T04:46:35.807137+0000","last_fullsized":"2024-03-26T04:46:35.807137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:20:17.582384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073429+0000","last_change":"2024-03-26T04:45:20.738821+0000","last_active":"2024-03-26T04:46:00.073429+0000","last_peered":"2024-03-26T04:46:00.073429+0000","last_clean":"2024-03-26T04:46:00.073429+0000","last_became_active":"2024-03-26T04:45:20.737911+0000","last_became_peered":"2024-03-26T04:45:20.737911+0000","last_unstale":"2024-03-26T04:46:00.073429+0000","last_undegraded":"2024-03-26T04:46:00.073429+0000","last_fullsized":"2024-03-26T04:46:00.073429+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:26:37.391492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074225+0000","last_change":"2024-03-26T04:45:22.760530+0000","last_active":"2024-03-26T04:46:00.074225+0000","last_peered":"2024-03-26T04:46:00.074225+0000","last_clean":"2024-03-26T04:46:00.074225+0000","last_became_active":"2024-03-26T04:45:22.760214+0000","last_became_peered":"2024-03-26T04:45:22.760214+0000","last_unstale":"2024-03-26T04:46:00.074225+0000","last_undegraded":"2024-03-26T04:46:00.074225+0000","last_fullsized":"2024-03-26T04:46:00.074225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:11:25.721050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076005+0000","last_change":"2024-03-26T04:45:26.776596+0000","last_active":"2024-03-26T04:46:00.076005+0000","last_peered":"2024-03-26T04:46:00.076005+0000","last_clean":"2024-03-26T04:46:00.076005+0000","last_became_active":"2024-03-26T04:45:26.776303+0000","last_became_peered":"2024-03-26T04:45:26.776303+0000","last_unstale":"2024-03-26T04:46:00.076005+0000","last_undegraded":"2024-03-26T04:46:00.076005+0000","last_fullsized":"2024-03-26T04:46:00.076005+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:18:15.114703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.806709+0000","last_change":"2024-03-26T04:45:24.776804+0000","last_active":"2024-03-26T04:46:35.806709+0000","last_peered":"2024-03-26T04:46:35.806709+0000","last_clean":"2024-03-26T04:46:35.806709+0000","last_became_active":"2024-03-26T04:45:24.776464+0000","last_became_peered":"2024-03-26T04:45:24.776464+0000","last_unstale":"2024-03-26T04:46:35.806709+0000","last_undegraded":"2024-03-26T04:46:35.806709+0000","last_fullsized":"2024-03-26T04:46:35.806709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:47:49.584903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075022+0000","last_change":"2024-03-26T04:45:20.741586+0000","last_active":"2024-03-26T04:46:00.075022+0000","last_peered":"2024-03-26T04:46:00.075022+0000","last_clean":"2024-03-26T04:46:00.075022+0000","last_became_active":"2024-03-26T04:45:20.741234+0000","last_became_peered":"2024-03-26T04:45:20.741234+0000","last_unstale":"2024-03-26T04:46:00.075022+0000","last_undegraded":"2024-03-26T04:46:00.075022+0000","last_fullsized":"2024-03-26T04:46:00.075022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:27:24.311362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.075357+0000","last_change":"2024-03-26T04:45:22.759704+0000","last_active":"2024-03-26T04:46:00.075357+0000","last_peered":"2024-03-26T04:46:00.075357+0000","last_clean":"2024-03-26T04:46:00.075357+0000","last_became_active":"2024-03-26T04:45:22.759452+0000","last_became_peered":"2024-03-26T04:45:22.759452+0000","last_unstale":"2024-03-26T04:46:00.075357+0000","last_undegraded":"2024-03-26T04:46:00.075357+0000","last_fullsized":"2024-03-26T04:46:00.075357+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:33:00.756040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076887+0000","last_change":"2024-03-26T04:45:26.784724+0000","last_active":"2024-03-26T04:46:00.076887+0000","last_peered":"2024-03-26T04:46:00.076887+0000","last_clean":"2024-03-26T04:46:00.076887+0000","last_became_active":"2024-03-26T04:45:26.784375+0000","last_became_peered":"2024-03-26T04:45:26.784375+0000","last_unstale":"2024-03-26T04:46:00.076887+0000","last_undegraded":"2024-03-26T04:46:00.076887+0000","last_fullsized":"2024-03-26T04:46:00.076887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:19:38.728539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.073771+0000","last_change":"2024-03-26T04:45:24.772992+0000","last_active":"2024-03-26T04:46:00.073771+0000","last_peered":"2024-03-26T04:46:00.073771+0000","last_clean":"2024-03-26T04:46:00.073771+0000","last_became_active":"2024-03-26T04:45:24.772653+0000","last_became_peered":"2024-03-26T04:45:24.772653+0000","last_unstale":"2024-03-26T04:46:00.073771+0000","last_undegraded":"2024-03-26T04:46:00.073771+0000","last_fullsized":"2024-03-26T04:46:00.073771+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:38.757217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469969+0000","last_change":"2024-03-26T04:45:20.743969+0000","last_active":"2024-03-26T04:46:00.469969+0000","last_peered":"2024-03-26T04:46:00.469969+0000","last_clean":"2024-03-26T04:46:00.469969+0000","last_became_active":"2024-03-26T04:45:20.743666+0000","last_became_peered":"2024-03-26T04:45:20.743666+0000","last_unstale":"2024-03-26T04:46:00.469969+0000","last_undegraded":"2024-03-26T04:46:00.469969+0000","last_fullsized":"2024-03-26T04:46:00.469969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:26:37.306783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.077244+0000","last_change":"2024-03-26T04:45:45.914102+0000","last_active":"2024-03-26T04:46:00.077244+0000","last_peered":"2024-03-26T04:46:00.077244+0000","last_clean":"2024-03-26T04:46:00.077244+0000","last_became_active":"2024-03-26T04:45:45.905675+0000","last_became_peered":"2024-03-26T04:45:45.905675+0000","last_unstale":"2024-03-26T04:46:00.077244+0000","last_undegraded":"2024-03-26T04:46:00.077244+0000","last_fullsized":"2024-03-26T04:46:00.077244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:27:08.036919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.471195+0000","last_change":"2024-03-26T04:45:26.783776+0000","last_active":"2024-03-26T04:46:00.471195+0000","last_peered":"2024-03-26T04:46:00.471195+0000","last_clean":"2024-03-26T04:46:00.471195+0000","last_became_active":"2024-03-26T04:45:26.783416+0000","last_became_peered":"2024-03-26T04:45:26.783416+0000","last_unstale":"2024-03-26T04:46:00.471195+0000","last_undegraded":"2024-03-26T04:46:00.471195+0000","last_fullsized":"2024-03-26T04:46:00.471195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:06:04.970363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076316+0000","last_change":"2024-03-26T04:45:24.777500+0000","last_active":"2024-03-26T04:46:00.076316+0000","last_peered":"2024-03-26T04:46:00.076316+0000","last_clean":"2024-03-26T04:46:00.076316+0000","last_became_active":"2024-03-26T04:45:24.777177+0000","last_became_peered":"2024-03-26T04:45:24.777177+0000","last_unstale":"2024-03-26T04:46:00.076316+0000","last_undegraded":"2024-03-26T04:46:00.076316+0000","last_fullsized":"2024-03-26T04:46:00.076316+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:45:33.144903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076617+0000","last_change":"2024-03-26T04:45:20.738638+0000","last_active":"2024-03-26T04:46:00.076617+0000","last_peered":"2024-03-26T04:46:00.076617+0000","last_clean":"2024-03-26T04:46:00.076617+0000","last_became_active":"2024-03-26T04:45:20.738332+0000","last_became_peered":"2024-03-26T04:45:20.738332+0000","last_unstale":"2024-03-26T04:46:00.076617+0000","last_undegraded":"2024-03-26T04:46:00.076617+0000","last_fullsized":"2024-03-26T04:46:00.076617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:53:00.385609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.470652+0000","last_change":"2024-03-26T04:45:22.758745+0000","last_active":"2024-03-26T04:46:00.470652+0000","last_peered":"2024-03-26T04:46:00.470652+0000","last_clean":"2024-03-26T04:46:00.470652+0000","last_became_active":"2024-03-26T04:45:22.758473+0000","last_became_peered":"2024-03-26T04:45:22.758473+0000","last_unstale":"2024-03-26T04:46:00.470652+0000","last_undegraded":"2024-03-26T04:46:00.470652+0000","last_fullsized":"2024-03-26T04:46:00.470652+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:18.388864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.198866+0000","last_change":"2024-03-26T04:45:26.788455+0000","last_active":"2024-03-26T04:46:00.198866+0000","last_peered":"2024-03-26T04:46:00.198866+0000","last_clean":"2024-03-26T04:46:00.198866+0000","last_became_active":"2024-03-26T04:45:26.788123+0000","last_became_peered":"2024-03-26T04:45:26.788123+0000","last_unstale":"2024-03-26T04:46:00.198866+0000","last_undegraded":"2024-03-26T04:46:00.198866+0000","last_fullsized":"2024-03-26T04:46:00.198866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:45:10.716106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076443+0000","last_change":"2024-03-26T04:45:24.773439+0000","last_active":"2024-03-26T04:46:00.076443+0000","last_peered":"2024-03-26T04:46:00.076443+0000","last_clean":"2024-03-26T04:46:00.076443+0000","last_became_active":"2024-03-26T04:45:24.773079+0000","last_became_peered":"2024-03-26T04:45:24.773079+0000","last_unstale":"2024-03-26T04:46:00.076443+0000","last_undegraded":"2024-03-26T04:46:00.076443+0000","last_fullsized":"2024-03-26T04:46:00.076443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:55:08.823253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.074080+0000","last_change":"2024-03-26T04:45:20.738319+0000","last_active":"2024-03-26T04:46:00.074080+0000","last_peered":"2024-03-26T04:46:00.074080+0000","last_clean":"2024-03-26T04:46:00.074080+0000","last_became_active":"2024-03-26T04:45:20.737734+0000","last_became_peered":"2024-03-26T04:45:20.737734+0000","last_unstale":"2024-03-26T04:46:00.074080+0000","last_undegraded":"2024-03-26T04:46:00.074080+0000","last_fullsized":"2024-03-26T04:46:00.074080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:34:03.766481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.199530+0000","last_change":"2024-03-26T04:45:22.756790+0000","last_active":"2024-03-26T04:46:00.199530+0000","last_peered":"2024-03-26T04:46:00.199530+0000","last_clean":"2024-03-26T04:46:00.199530+0000","last_became_active":"2024-03-26T04:45:22.756463+0000","last_became_peered":"2024-03-26T04:45:22.756463+0000","last_unstale":"2024-03-26T04:46:00.199530+0000","last_undegraded":"2024-03-26T04:46:00.199530+0000","last_fullsized":"2024-03-26T04:46:00.199530+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:55:08.853240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469287+0000","last_change":"2024-03-26T04:45:26.790222+0000","last_active":"2024-03-26T04:46:00.469287+0000","last_peered":"2024-03-26T04:46:00.469287+0000","last_clean":"2024-03-26T04:46:00.469287+0000","last_became_active":"2024-03-26T04:45:26.789912+0000","last_became_peered":"2024-03-26T04:45:26.789912+0000","last_unstale":"2024-03-26T04:46:00.469287+0000","last_undegraded":"2024-03-26T04:46:00.469287+0000","last_fullsized":"2024-03-26T04:46:00.469287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:25.756058+0000","last_clean_scrub_stamp":"2024-03-26T04:45:25.756058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:43.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.076687+0000","last_change":"2024-03-26T04:45:20.735677+0000","last_active":"2024-03-26T04:46:00.076687+0000","last_peered":"2024-03-26T04:46:00.076687+0000","last_clean":"2024-03-26T04:46:00.076687+0000","last_became_active":"2024-03-26T04:45:20.735355+0000","last_became_peered":"2024-03-26T04:45:20.735355+0000","last_unstale":"2024-03-26T04:46:00.076687+0000","last_undegraded":"2024-03-26T04:46:00.076687+0000","last_fullsized":"2024-03-26T04:46:00.076687+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:19.712048+0000","last_clean_scrub_stamp":"2024-03-26T04:45:19.712048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:38:31.372686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"67'27","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:00.469398+0000","last_change":"2024-03-26T04:45:22.752501+0000","last_active":"2024-03-26T04:46:00.469398+0000","last_peered":"2024-03-26T04:46:00.469398+0000","last_clean":"2024-03-26T04:46:00.469398+0000","last_became_active":"2024-03-26T04:45:22.752191+0000","last_became_peered":"2024-03-26T04:45:22.752191+0000","last_unstale":"2024-03-26T04:46:00.469398+0000","last_undegraded":"2024-03-26T04:46:00.469398+0000","last_fullsized":"2024-03-26T04:46:00.469398+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:21.723350+0000","last_clean_scrub_stamp":"2024-03-26T04:45:21.723350+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:16:49.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T04:46:35.807084+0000","last_change":"2024-03-26T04:45:24.780983+0000","last_active":"2024-03-26T04:46:35.807084+0000","last_peered":"2024-03-26T04:46:35.807084+0000","last_clean":"2024-03-26T04:46:35.807084+0000","last_became_active":"2024-03-26T04:45:24.780415+0000","last_became_peered":"2024-03-26T04:45:24.780415+0000","last_unstale":"2024-03-26T04:46:35.807084+0000","last_undegraded":"2024-03-26T04:46:35.807084+0000","last_fullsized":"2024-03-26T04:46:35.807084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:45:23.740867+0000","last_clean_scrub_stamp":"2024-03-26T04:45:23.740867+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:49:30.547179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32540,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712612,"statfs":{"total":95995035648,"available":95961714688,"internally_reserved":0,"allocated":1392640,"data_stored":923704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1380352,"data_stored":922634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":811008,"data_stored":333720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1400832,"data_stored":923844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":794624,"data_stored":332839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":823296,"data_stored":333106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":798720,"data_stored":332344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":786432,"data_stored":332619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:46:45.805 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T04:46:45.805 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T04:46:45.805 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T04:46:45.805 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T04:46:45.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:45 smithi012 ceph-mon[29863]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:45.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:45 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:45.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:45 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:45 smithi012 ceph-mon[41141]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:45 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:45 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:45 smithi186 ceph-mon[33438]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:45 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:45 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:46:46.837 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:46 smithi012 ceph-mon[29863]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:46:46.838 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:46 smithi012 ceph-mon[41141]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:46:46.838 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:46:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:46 smithi186 ceph-mon[33438]: from='client.24614 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:46:47.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 -- 172.21.15.12:0/21715633 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 msgr2=0x7f635c107470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:47.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 --2- 172.21.15.12:0/21715633 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c107470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f634c009990 tx=0x7f634c009ca0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 -- 172.21.15.12:0/21715633 shutdown_connections 2024-03-26T04:46:47.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 --2- 172.21.15.12:0/21715633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06b1c0 0x7f635c10ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 --2- 172.21.15.12:0/21715633 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c107470 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 --2- 172.21.15.12:0/21715633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f635c06a800 0x7f635c06ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 -- 172.21.15.12:0/21715633 >> 172.21.15.12:0/21715633 conn(0x7f635c077fe0 msgr2=0x7f635c0783f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 -- 172.21.15.12:0/21715633 shutdown_connections 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.649+0000 7f6362188700 1 -- 172.21.15.12:0/21715633 wait complete. 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 Processor -- start 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 -- start start 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 0x7f635c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f635c06b1c0 0x7f635c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c198040 con 0x7f635c06b1c0 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c198180 con 0x7f635c06a800 2024-03-26T04:46:47.653 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f6362188700 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c198340 con 0x7f635c107090 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f635b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 0x7f635c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f635b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 0x7f635c19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.12:44236/0 (socket says 172.21.15.12:44236) 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f635b7fe700 1 -- 172.21.15.12:0/148576226 learned_addr learned my addr 172.21.15.12:0/148576226 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.650+0000 7f635affd700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f635c06b1c0 0x7f635c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635bfff700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635b7fe700 1 -- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 msgr2=0x7f635c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635b7fe700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635b7fe700 1 -- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f635c06b1c0 msgr2=0x7f635c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635b7fe700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f635c06b1c0 0x7f635c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635b7fe700 1 -- 172.21.15.12:0/148576226 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350009520 con 0x7f635c06a800 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635bfff700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T04:46:47.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f635b7fe700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 0x7f635c19d820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f634c004760 tx=0x7f634c004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:47.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c01d070 con 0x7f635c06a800 2024-03-26T04:46:47.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f634c00bda0 con 0x7f635c06a800 2024-03-26T04:46:47.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c0095f0 con 0x7f635c06a800 2024-03-26T04:46:47.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c005280 con 0x7f635c06a800 2024-03-26T04:46:47.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.651+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c1a46f0 con 0x7f635c06a800 2024-03-26T04:46:47.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.652+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c068d40 con 0x7f635c06a800 2024-03-26T04:46:47.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.653+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f634c00b920 con 0x7f635c06a800 2024-03-26T04:46:47.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.654+0000 7f6358ff9700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f634406dac0 0x7f634406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T04:46:47.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.654+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7537+0+0 (secure 0 0 0) 0x7f634c08e8d0 con 0x7f635c06a800 2024-03-26T04:46:47.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.654+0000 7f635affd700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f634406dac0 0x7f634406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T04:46:47.656 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.654+0000 7f635affd700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f634406dac0 0x7f634406ff80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f635c078880 tx=0x7f6350009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T04:46:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.661+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f634c057ca0 con 0x7f635c06a800 2024-03-26T04:46:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:47 smithi012 ceph-mon[29863]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:47 smithi012 ceph-mon[41141]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:47.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.908+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f635c111660 con 0x7f635c06a800 2024-03-26T04:46:47.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.909+0000 7f6358ff9700 1 -- 172.21.15.12:0/148576226 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f634c05b360 con 0x7f635c06a800 2024-03-26T04:46:47.909 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-03-26T04:46:47.909 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T04:46:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f634406dac0 msgr2=0x7f634406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f634406dac0 0x7f634406ff80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f635c078880 tx=0x7f6350009260 comp rx=0 tx=0).stop 2024-03-26T04:46:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 msgr2=0x7f635c19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:46:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 0x7f635c19d820 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f634c004760 tx=0x7f634c004790 comp rx=0 tx=0).stop 2024-03-26T04:46:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 shutdown_connections 2024-03-26T04:46:47.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f634406dac0 0x7f634406ff80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f635c107090 0x7f635c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f635c06a800 0x7f635c19d820 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 --2- 172.21.15.12:0/148576226 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f635c06b1c0 0x7f635c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T04:46:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 >> 172.21.15.12:0/148576226 conn(0x7f635c077fe0 msgr2=0x7f635c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:46:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 shutdown_connections 2024-03-26T04:46:47.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-03-26T04:46:47.911+0000 7f6362188700 1 -- 172.21.15.12:0/148576226 wait complete. 2024-03-26T04:46:47.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:47 smithi186 ceph-mon[33438]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:48.652 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T04:46:48.652 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T04:46:48.652 INFO:teuthology.run_tasks:Running task workunit... 2024-03-26T04:46:48.666 INFO:tasks.workunit:Pulling workunits from ref 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T04:46:48.666 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T04:46:48.667 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T04:46:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:46:48.684 INFO:teuthology.orchestra.run.smithi012.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-26T04:46:48.684 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T04:46:48.741 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-26T04:46:48.742 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-26T04:46:48.797 INFO:tasks.workunit:timeout=3h 2024-03-26T04:46:48.797 INFO:tasks.workunit:cleanup=True 2024-03-26T04:46:48.798 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T04:46:48.852 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T04:46:48.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/148576226' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:46:48.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/148576226' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:46:48.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/148576226' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:46:49.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:49 smithi186 ceph-mon[33438]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:49 smithi012 ceph-mon[29863]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:50.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:49 smithi012 ceph-mon[41141]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:51.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:51 smithi186 ceph-mon[33438]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:51.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:46:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:46:52.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:51 smithi012 ceph-mon[29863]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:52.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:51 smithi012 ceph-mon[41141]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:52.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:52 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:52.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:52 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:52.906 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:46:52 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:46:52.523Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002264783s 2024-03-26T04:46:52.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:52 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:53.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:53 smithi186 ceph-mon[33438]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:53 smithi012 ceph-mon[41141]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:53 smithi012 ceph-mon[29863]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:55.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:55 smithi186 ceph-mon[33438]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:55 smithi012 ceph-mon[29863]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:55 smithi012 ceph-mon[41141]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:57.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:46:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:46:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:57.214 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 podman[64205]: 2024-03-26 04:46:57.148938085 +0000 UTC m=+0.134878158 container create c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=ubi8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.buildah.version=1.32.0, vendor=Red Hat, Inc., version=8.5, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897) 2024-03-26T04:46:57.520 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 podman[64205]: 2024-03-26 04:46:57.367848678 +0000 UTC m=+0.353788752 container init c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.buildah.version=1.32.0, maintainer=Paul Cuzner , release=239.1651231664, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git) 2024-03-26T04:46:57.520 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 podman[64205]: 2024-03-26 04:46:57.371065939 +0000 UTC m=+0.357006015 container start c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a, description=Ceph Grafana Container, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, version=8.5, com.redhat.component=ubi8-container) 2024-03-26T04:46:57.520 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 bash[64205]: c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 2024-03-26T04:46:57.520 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 systemd[1]: Started Ceph grafana.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:46:57.771 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-mon[33438]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:57.771 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:57.771 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:57.771 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:57.771 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.5463372Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.54653971Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546559183Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546567206Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546573258Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546579219Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546584812Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546590587Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546596327Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546601772Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T04:46:57.772 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546612358Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546617611Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546623131Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=settings t=2024-03-26T04:46:57.546639081Z level=info msg="App mode production" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=sqlstore t=2024-03-26T04:46:57.546688107Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=sqlstore t=2024-03-26T04:46:57.546704136Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.547535555Z level=info msg="Starting DB migrations" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.547615293Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.548157194Z level=info msg="Executing migration" id="create user table" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.548574055Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.549058438Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.54947345Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.55001054Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T04:46:57.773 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.550414122Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.551127015Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.551530392Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.551945842Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.552387132Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.552711719Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.553076513Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.553479409Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.553795527Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.554218358Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.554558376Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.555002153Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.555416898Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.556281292Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T04:46:57.774 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.557435648Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.55832298Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.559161355Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.560004995Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.560851665Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.561273072Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.562092704Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.562520447Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.564874382Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.565399073Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.568217186Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.571053777Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.573747381Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.574195639Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T04:46:57.775 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.576866634Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T04:46:57.776 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.577283561Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T04:46:57.776 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.57970911Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T04:46:57.776 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.582431085Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T04:46:57.776 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.585221386Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T04:46:57.776 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.588086185Z level=info msg="Executing migration" id="create star table" 2024-03-26T04:46:57.776 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.590936526Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T04:46:57.777 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.593702066Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T04:46:57.777 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.596641494Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T04:46:57.777 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.599368887Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T04:46:57.777 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.602035801Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T04:46:57.777 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.604692224Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.607432027Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.610114376Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.612796223Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.615505637Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.618318645Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.621049069Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.623737345Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.626355794Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.629174713Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.631864257Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.634625823Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.637185064Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T04:46:57.778 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.639861713Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.642634353Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.645421156Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.647864273Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.648700832Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.649556837Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.65041135Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.651288058Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.652109705Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.65295902Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.653840909Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.65471424Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.655559713Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.65640784Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T04:46:57.779 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.657199932Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.657932168Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.658645443Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.659482562Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.660336758Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.661190625Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.662034596Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.662888803Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.66375717Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.664703175Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.665510477Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.666809996Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.667685679Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.668501468Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T04:46:57.780 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.669355195Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.670202399Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.671059986Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.671913792Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.672791346Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.673706683Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.674567423Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.675407971Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.676288265Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.677149882Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.678011317Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.678457029Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.68133094Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T04:46:57.781 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.684166572Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.685252843Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.687964656Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.69076176Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.6936044Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.694070507Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.696982239Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.697556097Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.699759313Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.702553264Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.703394386Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.706372323Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.70922947Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T04:46:57.782 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.712105784Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.712586052Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.715365322Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.718020778Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.720620434Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.72312894Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.725790301Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.728383891Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.731089148Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.733812727Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.736518539Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.739267932Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.742050467Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.744742846Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T04:46:57.783 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.7474636Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.750015386Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.752766336Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.755518472Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.758301766Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.760807134Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.761767452Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.762679315Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.763524614Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.764384568Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.765218655Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.765828951Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.76671518Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.767580874Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T04:46:57.784 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.768377371Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T04:46:57.785 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.769224974Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T04:46:57.785 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.770062961Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T04:46:57.785 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.770887369Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T04:46:57.785 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.771749134Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T04:46:57.785 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.774338887Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T04:46:58.024 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.777019701Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T04:46:58.024 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.77975239Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T04:46:58.024 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.782403733Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T04:46:58.024 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.785213258Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T04:46:58.024 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.785648243Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.788388291Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.79117481Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.793876248Z level=info msg="Executing migration" id="create session table" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.794353357Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.794667376Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.797057616Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.797504194Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.800255207Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.802960996Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.803304978Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.803588572Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.803911289Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.804424131Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T04:46:58.025 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.804697231Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.807132396Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.809824839Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.810467971Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.8132144Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.815880765Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.816335655Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.819024294Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.821724853Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.824405605Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.826996383Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.827470877Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.82785074Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.828266927Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T04:46:58.026 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.828641877Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T04:46:58.027 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.830051969Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T04:46:58.027 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.830452889Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T04:46:58.027 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.830884305Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T04:46:58.027 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.831196929Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T04:46:58.027 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.831538768Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.831923434Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.832549344Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.833256605Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.833880842Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.834514762Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.834938976Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T04:46:58.028 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.835219169Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.835482836Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.835895741Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.836340717Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.836688175Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.83710125Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.837482967Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.838132558Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.838751556Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.839065022Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.839477225Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.839883083Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.840520048Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.840838136Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.841282238Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.841706848Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T04:46:58.029 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.842032031Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.842453598Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.842867347Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.843292363Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.843700865Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.84515945Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.847876284Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.850419809Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.851083362Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.853832562Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.856513231Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T04:46:58.030 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.859066769Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.86172222Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.864299364Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.867069938Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.869624679Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.872313032Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.875062753Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.877788221Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.880395177Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T04:46:58.034 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.883074557Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.885737843Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.886484393Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.887318149Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.888119105Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.888902903Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.889339357Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.890121944Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.890509941Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.890923066Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.891710111Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.892135325Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.892944462Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.893367734Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.893808507Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.894621483Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T04:46:58.035 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.895227792Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.89561139Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.896452975Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.897260423Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.897585612Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.898378804Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.899209445Z level=info msg="Executing migration" id="create team table" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.899606078Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.900086523Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.900513146Z level=info msg="Executing migration" id="create team member table" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.901323265Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.902166954Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.903020195Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.903505013Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.904347407Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.905199188Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T04:46:58.036 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.906052769Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.906458305Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.907305461Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.908165545Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.908775916Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.909285566Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.910124344Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.912842845Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.915574779Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.918347163Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.921044836Z level=info msg="Executing migration" id="create tag table" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.923843889Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.926588836Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.929301656Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.932040988Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T04:46:58.037 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.934896761Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.937763147Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.93823035Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.93904803Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.939857311Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.940738575Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.941572541Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.942436035Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.943258285Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.944040346Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.944788707Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.945658072Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.94651968Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.947398714Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.948226497Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T04:46:58.038 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.949090571Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.949943186Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.950792005Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.951708997Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.952552875Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.953386367Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.954260229Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.955129904Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.955987556Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.956836444Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.957717419Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.958562872Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.95942138Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.960314167Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T04:46:58.039 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.961216642Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.962068782Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.962885914Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.963740484Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.96459382Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.967367303Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.970214496Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.97311273Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.975938878Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.978616107Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.981364494Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.984266171Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.986950987Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.989733336Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.992615537Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.995437142Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T04:46:58.040 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:57.998178574Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.000972559Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.003745873Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.006508762Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.009374567Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.012868316Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.016292279Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.018997189Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.021809713Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.024524751Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T04:46:58.041 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.027215979Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T04:46:58.101 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[29863]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:58.101 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[41141]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.102 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.029910964Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.032700791Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.035388202Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.038123138Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.040682465Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.043424458Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.046167422Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.048788927Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.051565013Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.054253984Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.057058154Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T04:46:58.275 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.059753719Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.062491839Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.063362915Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.064261225Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.065118653Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.065925208Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.066753137Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.06761042Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T04:46:58.276 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.068501485Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.069394257Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.070297815Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.07113807Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.072103462Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.072902769Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.073767254Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.074604909Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.075487907Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.076456742Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.077264747Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.078138944Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.078980798Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T04:46:58.277 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.07986219Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.080731035Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.081605944Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.082492297Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.08332855Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.084188771Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.085049341Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.085875242Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.086737907Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.086826688Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.087582424Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.088464668Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.089340046Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.08946944Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T04:46:58.278 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.089585538Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.090187756Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.091034253Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.09193031Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.092772013Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.095616386Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.098429337Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T04:46:58.279 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.101189363Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T04:46:58.280 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.103935844Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T04:46:58.280 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.106706876Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T04:46:58.280 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.109472948Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T04:46:58.280 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.113827295Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.116492984Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.119399825Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.123653958Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.127959217Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.130794204Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.133592325Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.136470045Z level=info msg="Executing migration" id="create permission table" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.13921161Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.142085294Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.144867497Z level=info msg="Executing migration" id="create role table" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.147626013Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.150306788Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.153077734Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T04:46:58.281 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.15583301Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.158541089Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.161204604Z level=info msg="Executing migration" id="create team role table" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.16393341Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.166711105Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.169390768Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.172147193Z level=info msg="Executing migration" id="create user role table" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.172559022Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.175307808Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.178038005Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.180747409Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.183487998Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.186096436Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T04:46:58.282 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.188807688Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.189781275Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.191528302Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.193938914Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.194415487Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.194876975Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.197583197Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.198070125Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.199049621Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.199450537Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.199969034Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.20027712Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.200667901Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.201111591Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T04:46:58.283 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.20145063Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.201736543Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.202087529Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.202470778Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.202925991Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.203888483Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.204187573Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.204594956Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.205026019Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.205455785Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.206427782Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.206878055Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.209441371Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.20954293Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:46:58.284 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.209872822Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.209970515Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.21025927Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.210625192Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.211049688Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.211486619Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.211931408Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.212347116Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.212774863Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.213159648Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.213549614Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.213987325Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:46:58.285 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.214438454Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.217082588Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.220068783Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.221135547Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.222132389Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.222450881Z level=info msg="Executing migration" id="add share column" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.223452803Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.223896687Z level=info msg="Executing migration" id="create file table" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.224326109Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.226953879Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.227466903Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.230121073Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.232884583Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T04:46:58.286 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.235456056Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.238135949Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.240882221Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.243609765Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.246293497Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.248956984Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.25161301Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.252379549Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.253138893Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.253527737Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.254589855Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.255657303Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.25642979Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.266294718Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T04:46:58.287 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.267076196Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.267807374Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.27122308Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.271986792Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.272772301Z level=info msg="Executing migration" id="create folder table" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.273557397Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.274349927Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.275126988Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.276004304Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=migrator t=2024-03-26T04:46:58.276704802Z level=info msg="migrations completed" performed=485 skipped=0 duration=729.117566ms 2024-03-26T04:46:58.288 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=sqlstore t=2024-03-26T04:46:58.277149168Z level=info msg="Created default organization" 2024-03-26T04:46:58.529 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=plugin.loader t=2024-03-26T04:46:58.335884436Z level=info msg="Plugin registered" pluginID=input 2024-03-26T04:46:58.529 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=plugin.loader t=2024-03-26T04:46:58.364162244Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T04:46:58.529 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=plugin.loader t=2024-03-26T04:46:58.364277883Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T04:46:58.529 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=secrets t=2024-03-26T04:46:58.364378454Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T04:46:58.529 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=query_data t=2024-03-26T04:46:58.367654879Z level=info msg="Query Service initialization" 2024-03-26T04:46:58.529 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=live.push_http t=2024-03-26T04:46:58.370436876Z level=info msg="Live Push Gateway initialization" 2024-03-26T04:46:58.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:58 smithi012 ceph-mon[29863]: Deploying daemon node-exporter.a on smithi012 2024-03-26T04:46:58.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:58 smithi012 ceph-mon[41141]: Deploying daemon node-exporter.a on smithi012 2024-03-26T04:46:58.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-mon[33438]: Deploying daemon node-exporter.a on smithi012 2024-03-26T04:46:59.405 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:46:59 smithi012 bash[76837]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=infra.usagestats.collector t=2024-03-26T04:46:58.995741841Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=provisioning.datasources t=2024-03-26T04:46:58.99675306Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=provisioning.datasources t=2024-03-26T04:46:59.007467346Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=provisioning.alerting t=2024-03-26T04:46:59.018004425Z level=info msg="starting to provision alerting" 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=provisioning.alerting t=2024-03-26T04:46:59.01802239Z level=info msg="finished to provision alerting" 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=ngalert.state.manager t=2024-03-26T04:46:59.018095612Z level=info msg="Warming state cache for startup" 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=ngalert.state.manager t=2024-03-26T04:46:59.018234594Z level=info msg="State cache has been initialized" states=0 duration=138.344µs 2024-03-26T04:46:59.464 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=grafanaStorageLogger t=2024-03-26T04:46:59.018230739Z level=info msg="storage starting" 2024-03-26T04:46:59.465 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=ticker t=2024-03-26T04:46:59.018315164Z level=info msg=starting first_tick=2024-03-26T04:47:00Z 2024-03-26T04:46:59.465 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=ngalert.multiorg.alertmanager t=2024-03-26T04:46:59.018340606Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T04:46:59.465 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=http.server t=2024-03-26T04:46:59.020150996Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T04:46:59.465 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=sqlstore.transactions t=2024-03-26T04:46:59.047078849Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:46:59.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:46:59 smithi186 ceph-mon[33438]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:00.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:46:59 smithi012 ceph-mon[29863]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:00.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:46:59 smithi012 ceph-mon[41141]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:00.905 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:00 smithi012 bash[76837]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:47:00.905 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:00 smithi012 bash[76837]: Writing manifest to image destination 2024-03-26T04:47:01.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-mon[29863]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:01.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:01.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:01.308 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-mon[41141]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:01.308 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:01.308 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:01.308 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 podman[76837]: 2024-03-26 04:47:00.998664383 +0000 UTC m=+2.406142494 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:47:01.308 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 podman[76837]: 2024-03-26 04:47:01.080435113 +0000 UTC m=+2.487913224 container create 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:01.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:01 smithi186 ceph-mon[33438]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:01.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:01 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:01.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:01.655 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 podman[76837]: 2024-03-26 04:47:01.308023854 +0000 UTC m=+2.715501965 container init 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 podman[76837]: 2024-03-26 04:47:01.310785447 +0000 UTC m=+2.718263575 container start 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.315Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.315Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.315Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.315Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.315Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:47:01.656 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:47:01.657 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:47:01.658 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:47:01.658 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:47:01.658 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:47:01.658 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:47:01.658 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:47:01.658 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:47:01.659 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:47:01.660 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:47:01.660 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:47:01.660 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:47:01.661 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:47:01.662 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:47:01.662 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:47:01.662 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:47:01.662 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:47:01.663 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:47:01.663 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:47:01.663 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a[76889]: ts=2024-03-26T04:47:01.316Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:47:01.663 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 bash[76837]: 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 2024-03-26T04:47:01.663 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 04:47:01 smithi012 systemd[1]: Started Ceph node-exporter.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:47:02.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:47:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:47:02.629 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.629 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.629 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.629 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:02 smithi186 ceph-mon[33438]: Deploying daemon node-exporter.b on smithi186 2024-03-26T04:47:02.629 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:02 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:02.630 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:02 smithi186 systemd[1]: Starting Ceph node-exporter.b for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:47:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[29863]: Deploying daemon node-exporter.b on smithi186 2024-03-26T04:47:02.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:02.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:02.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[41141]: Deploying daemon node-exporter.b on smithi186 2024-03-26T04:47:02.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:02 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:02.963 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:02 smithi186 bash[64482]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:47:03.461 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:03 smithi186 bash[64482]: Getting image source signatures 2024-03-26T04:47:03.461 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:03 smithi186 bash[64482]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:47:03.462 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:03 smithi186 bash[64482]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:47:03.462 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:03 smithi186 bash[64482]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:47:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:03 smithi186 ceph-mon[33438]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:03 smithi012 ceph-mon[29863]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:03.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:03 smithi012 ceph-mon[41141]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:04.963 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:04 smithi186 bash[64482]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:47:04.963 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:04 smithi186 bash[64482]: Writing manifest to image destination 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 podman[64482]: 2024-03-26 04:47:05.000098607 +0000 UTC m=+2.462489886 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 podman[64482]: 2024-03-26 04:47:05.119791802 +0000 UTC m=+2.582183080 container create 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 podman[64482]: 2024-03-26 04:47:05.301313984 +0000 UTC m=+2.763705267 container init 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 podman[64482]: 2024-03-26 04:47:05.30431305 +0000 UTC m=+2.766704337 container start 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.308Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.308Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.308Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.308Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:47:05.378 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:47:05.379 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:47:05.380 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:47:05.381 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:47:05.382 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:47:05.382 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:47:05.382 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b[64532]: ts=2024-03-26T04:47:05.309Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:47:05.382 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 bash[64482]: 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 2024-03-26T04:47:05.713 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 04:47:05 smithi186 systemd[1]: Started Ceph node-exporter.b for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:47:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-mon[33438]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:05.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:05.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:05 smithi012 ceph-mon[29863]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:05.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:05.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:05.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:05 smithi012 ceph-mon[41141]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:05.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:05.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:06 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:06 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:06 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:06 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:06.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:06.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:47:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:06.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:06.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:06 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:07.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:07 smithi012 ceph-mon[29863]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:07.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:07 smithi012 ceph-mon[41141]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:07.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:07 smithi186 ceph-mon[33438]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:09.855 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:09 smithi186 ceph-mon[33438]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:09.855 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:09 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:09.855 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:09 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:09.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:09 smithi012 ceph-mon[29863]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:09.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:09 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:09.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:09 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:09.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:09 smithi012 ceph-mon[41141]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:09.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:09 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:09.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:09 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:11.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[29863]: Reconfiguring daemon alertmanager.a on smithi012 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:11.338 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.339 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:47:11.339 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:11 smithi012 ceph-mon[41141]: Reconfiguring daemon alertmanager.a on smithi012 2024-03-26T04:47:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:47:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-mon[33438]: Reconfiguring daemon alertmanager.a on smithi012 2024-03-26T04:47:12.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:12 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:12.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:47:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:47:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:12 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:12 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:13.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:13 smithi012 ceph-mon[29863]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:13.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:13 smithi012 ceph-mon[41141]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:13.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:13 smithi186 ceph-mon[33438]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:14.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:13 smithi012 systemd[1]: Stopping Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:47:14.481 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:14 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[76178]: ts=2024-03-26T04:47:14.163Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:47:14.481 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:14 smithi012 podman[77602]: 2024-03-26 04:47:14.184194897 +0000 UTC m=+0.129957002 container died 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:14.905 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:14 smithi012 podman[77602]: 2024-03-26 04:47:14.689369897 +0000 UTC m=+0.635132011 container cleanup 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:14.905 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:14 smithi012 bash[77602]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:47:15.166 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-mon[29863]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:15.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-mon[41141]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:15.167 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:14 smithi012 podman[77615]: 2024-03-26 04:47:14.915980021 +0000 UTC m=+0.731417071 container remove 5e775bead032bbeb45cb82a3a051ed006fa525a2b48d6ad3c30409c014a86b23 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:15.167 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 podman[77615]: 2024-03-26 04:47:15.016842911 +0000 UTC m=+0.832279963 volume remove bdd23dc1eb1bd84593024465bf288bd45669a5d4d2d2351c6ef4d033439ac979 2024-03-26T04:47:15.167 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:47:15.167 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 systemd[1]: Stopped Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:47:15.447 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 systemd[1]: Starting Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:47:15.447 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 podman[77711]: 2024-03-26 04:47:15.376147136 +0000 UTC m=+0.107900161 volume create 60636a367c1d1050175b38790ce272cfccb84f26c90b9115b93c3caa83a574e1 2024-03-26T04:47:15.447 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 podman[77711]: 2024-03-26 04:47:15.28645358 +0000 UTC m=+0.018206617 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T04:47:15.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:15 smithi186 ceph-mon[33438]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:15.705 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 podman[77711]: 2024-03-26 04:47:15.468278802 +0000 UTC m=+0.200031827 container create 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:16.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 podman[77711]: 2024-03-26 04:47:15.805446576 +0000 UTC m=+0.537199620 container init 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:16.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 podman[77711]: 2024-03-26 04:47:15.808603206 +0000 UTC m=+0.540356242 container start 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:16.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.821Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:47:16.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.821Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.822Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.12 port=9094 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.824Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.853Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.853Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.856Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:15.856Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 bash[77711]: 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 2024-03-26T04:47:16.156 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:15 smithi012 systemd[1]: Started Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:47:16.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:16 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:16.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:16 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:16.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:16 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:16.464 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:16.464 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:16.464 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:16.464 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:16.464 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:16.464 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:16.905 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:47:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:18.132 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:17 smithi186 ceph-mon[33438]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:47:18.132 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:17 smithi186 ceph-mon[33438]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:18.132 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:17 smithi186 ceph-mon[33438]: Reconfiguring daemon prometheus.a on smithi186 2024-03-26T04:47:18.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-mon[29863]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:47:18.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-mon[29863]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:18.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-mon[29863]: Reconfiguring daemon prometheus.a on smithi186 2024-03-26T04:47:18.155 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:17.825Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000884538s 2024-03-26T04:47:18.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-mon[41141]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:47:18.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-mon[41141]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:18.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:17 smithi012 ceph-mon[41141]: Reconfiguring daemon prometheus.a on smithi186 2024-03-26T04:47:19.463 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 systemd[1]: Stopping Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:47:19.955 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.634Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:47:19.955 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:47:19.955 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:47:19.955 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:47:19.955 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[62249]: ts=2024-03-26T04:47:19.635Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T04:47:19.956 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:19 smithi186 podman[65439]: 2024-03-26 04:47:19.645330797 +0000 UTC m=+0.137415671 container died 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:20.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:19 smithi186 ceph-mon[33438]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:20.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:19 smithi012 ceph-mon[29863]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:20.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:19 smithi012 ceph-mon[41141]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:21.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:20 smithi186 ceph-mon[33438]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:21.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:20 smithi012 ceph-mon[29863]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:21.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:20 smithi012 ceph-mon[41141]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:21 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:22.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:47:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:47:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:21 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:22.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:21 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:22.963 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:22 smithi186 podman[65439]: 2024-03-26 04:47:22.578490796 +0000 UTC m=+3.070575667 container cleanup 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:22.963 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:22 smithi186 bash[65439]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:47:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:22 smithi012 ceph-mon[29863]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:23.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:22 smithi012 ceph-mon[41141]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:23.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:22 smithi186 ceph-mon[33438]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:24.902 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:24 smithi186 podman[65453]: 2024-03-26 04:47:24.469583836 +0000 UTC m=+4.826970022 container remove 417d9d877f6356112b39552bba0aee7eead64d0a117d0065ed789dc30cadb270 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:24.902 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:24 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T04:47:24.902 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:24 smithi186 systemd[1]: Stopped Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:47:24.902 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:24 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.094s CPU time. 2024-03-26T04:47:24.902 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:24 smithi186 systemd[1]: Starting Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T04:47:25.158 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:24 smithi186 podman[65552]: 2024-03-26 04:47:24.806711976 +0000 UTC m=+0.018924635 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T04:47:25.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:25 smithi186 ceph-mon[33438]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:25.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:25 smithi012 ceph-mon[41141]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:25 smithi012 ceph-mon[29863]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:26.150 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:25 smithi186 podman[65552]: 2024-03-26 04:47:25.887237297 +0000 UTC m=+1.099449999 container create a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:26.150 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 podman[65552]: 2024-03-26 04:47:26.122737998 +0000 UTC m=+1.334950656 container init a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:26.150 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 podman[65552]: 2024-03-26 04:47:26.125556205 +0000 UTC m=+1.337768871 container start a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:47:26.154 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 04:47:25 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T04:47:25.828Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00405584s 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.151Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.151Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.151Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi186 (none))" 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.151Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.151Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.152Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.153Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T04:47:26.426 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.154Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.154Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.156Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.156Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.889µs 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.156Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.157Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.157Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.157Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.834µs wal_replay_duration=681.458µs wbl_replay_duration=107ns total_replay_duration=726.035µs 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.158Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.158Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.158Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.179Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.46869ms db_storage=1.189µs remote_storage=1.267µs web_handler=420ns query_engine=660ns scrape=388.062µs scrape_sd=73.964µs notify=13.655µs notify_sd=17.189µs rules=19.751654ms tracing=6.903µs 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.179Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T04:47:26.179Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T04:47:26.427 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 bash[65552]: a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 2024-03-26T04:47:26.428 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 04:47:26 smithi186 systemd[1]: Started Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T04:47:26.533 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STOPPING 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STOPPED 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STARTING 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Serving on http://:::9283 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STARTED 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STOPPING 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STOPPED 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STARTING 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Serving on http://:::9283 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STARTED 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STOPPING 2024-03-26T04:47:26.534 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:47:26.535 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STOPPED 2024-03-26T04:47:26.535 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STARTING 2024-03-26T04:47:26.905 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Serving on http://:::9283 2024-03-26T04:47:26.905 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:04:47:26] ENGINE Bus STARTED 2024-03-26T04:47:27.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:27.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:47:27.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:47:27.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi012.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi012.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:47:27.433 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:47:27.434 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.434 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:27 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:47:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:47:27.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi012.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi012.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi012.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi012.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:47:27.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:27.658 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:27 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:29.633 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:29 smithi186 ceph-mon[33438]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:29.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:29 smithi012 ceph-mon[29863]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:29.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:29 smithi012 ceph-mon[41141]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:31.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[29863]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:31.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:31.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:31.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:31.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[41141]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:31.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:31.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:31.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:31 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:31.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:31 smithi186 ceph-mon[33438]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:31.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:31 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:31.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:31 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:31.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:31 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:32.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:47:31 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:47:32.531 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:32 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:32.532 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:32 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:32 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:33.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:33 smithi012 ceph-mon[29863]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:33.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:33 smithi012 ceph-mon[41141]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:33 smithi186 ceph-mon[33438]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:34.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:34.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:34.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:34.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:34.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:34 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:34 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:34 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:34.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:34 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:47:35.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:35 smithi012 ceph-mon[41141]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:35.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:35 smithi012 ceph-mon[29863]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:35 smithi186 ceph-mon[33438]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:37.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:47:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:37.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:37 smithi186 ceph-mon[33438]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:38.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:37 smithi012 ceph-mon[29863]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:38.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:37 smithi012 ceph-mon[41141]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:39.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:39 smithi186 ceph-mon[33438]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:40.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:39 smithi012 ceph-mon[29863]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:40.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:39 smithi012 ceph-mon[41141]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:41.939 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:41 smithi186 ceph-mon[33438]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:42.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:41 smithi012 ceph-mon[29863]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:42.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:41 smithi012 ceph-mon[41141]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:42.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:47:41 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:47:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:42 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:43.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:42 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:43.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:42 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:43 smithi186 ceph-mon[33438]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:44.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:43 smithi012 ceph-mon[29863]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:44.056 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:43 smithi012 ceph-mon[41141]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:45.744 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:45 smithi012 ceph-mon[41141]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:45.744 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:45 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:45.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:45 smithi012 ceph-mon[29863]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:45.744 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:45 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:45 smithi186 ceph-mon[33438]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:45 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:47.155 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:47:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:47.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:47 smithi186 ceph-mon[33438]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:47 smithi012 ceph-mon[29863]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:48.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:47 smithi012 ceph-mon[41141]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:49.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:49 smithi186 ceph-mon[33438]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:49 smithi012 ceph-mon[29863]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:50.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:49 smithi012 ceph-mon[41141]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:51.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:51 smithi186 ceph-mon[33438]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:52.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:51 smithi012 ceph-mon[29863]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:52.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:51 smithi012 ceph-mon[41141]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:52.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:47:51 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:47:52.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:52 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:53.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:52 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:53.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:52 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:53.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:53 smithi186 ceph-mon[33438]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:53 smithi012 ceph-mon[29863]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:53 smithi012 ceph-mon[41141]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:55.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:55 smithi186 ceph-mon[33438]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:55 smithi012 ceph-mon[29863]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:55 smithi012 ceph-mon[41141]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:56.267 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to '9f04c91b77e4619963d41730d8cf756e3a6152de'. 2024-03-26T04:47:56.267 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.268 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T04:47:56.269 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-03-26T04:47:56.269 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at 9f04c91b77e Merge PR #56391 into squid 2024-03-26T04:47:56.272 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T04:47:56.293 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T04:47:56.296 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T04:47:56.296 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T04:47:57.052 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T04:47:57.125 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T04:47:57.154 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:47:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:47:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:57.165 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T04:47:57.167 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T04:47:57.167 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T04:47:57.205 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T04:47:57.209 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-03-26T04:47:57.209 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T04:47:57.265 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-26T04:47:57.266 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-26T04:47:57.266 DEBUG:teuthology.orchestra.run.smithi012:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-26T04:47:57.332 INFO:tasks.workunit.client.0.smithi012.stderr:+ parallel=1 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' '' = --serial ']' 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ crimson=0 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' '' = --crimson ']' 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ color= 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -t 1 ']' 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T04:47:57.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T04:47:57.334 INFO:tasks.workunit.client.0.smithi012.stderr:+ declare -A pids 2024-03-26T04:47:57.334 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.334 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.334 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_aio 2024-03-26T04:47:57.334 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_aio' 2024-03-26T04:47:57.334 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_aio 2024-03-26T04:47:57.335 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.335 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_aio 2024-03-26T04:47:57.335 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78483 2024-03-26T04:47:57.335 INFO:tasks.workunit.client.0.smithi012.stdout:test api_aio on pid 78483 2024-03-26T04:47:57.338 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_aio on pid 78483' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78483 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_aio_pp 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_aio_pp' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_aio_pp 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.339 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.340 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_aio_pp 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78490 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_aio_pp on pid 78490' 2024-03-26T04:47:57.341 INFO:tasks.workunit.client.0.smithi012.stdout:test api_aio_pp on pid 78490 2024-03-26T04:47:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78490 2024-03-26T04:47:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-26T04:47:57.345 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_io 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_io' 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_io 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.346 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.347 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.348 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_io 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78499 2024-03-26T04:47:57.349 INFO:tasks.workunit.client.0.smithi012.stdout:test api_io on pid 78499 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_io on pid 78499' 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78499 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_io_pp 2024-03-26T04:47:57.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_io_pp' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_io_pp 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.354 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.355 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.356 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_io_pp 2024-03-26T04:47:57.357 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78510 2024-03-26T04:47:57.358 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_io_pp on pid 78510' 2024-03-26T04:47:57.358 INFO:tasks.workunit.client.0.smithi012.stdout:test api_io_pp on pid 78510 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78510 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_asio 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_asio' 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_asio 2024-03-26T04:47:57.363 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.364 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.366 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.367 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.368 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_asio 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.369 INFO:tasks.workunit.client.0.smithi012.stdout:test api_asio on pid 78526 2024-03-26T04:47:57.376 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78526 2024-03-26T04:47:57.376 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_asio on pid 78526' 2024-03-26T04:47:57.376 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78526 2024-03-26T04:47:57.376 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.376 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.376 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_list 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_list' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.377 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_list 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.379 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.380 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.381 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.382 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78483/exe 2024-03-26T04:47:57.383 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_list 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78544 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_list on pid 78544' 2024-03-26T04:47:57.384 INFO:tasks.workunit.client.0.smithi012.stdout:test api_list on pid 78544 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78544 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.391 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_lock 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_lock' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.392 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.393 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_lock 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.395 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_aio: /' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78490/exe 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.396 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.397 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.398 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_lock 2024-03-26T04:47:57.399 INFO:tasks.workunit.client.0.smithi012.stdout:test api_lock on pid 78563 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78563 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_lock on pid 78563' 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78563 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_lock_pp 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_lock_pp' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.404 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_lock_pp 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-26T04:47:57.405 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.406 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78499/exe 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.407 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.409 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.410 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.411 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_io: /' 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-26T04:47:57.412 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_lock_pp 2024-03-26T04:47:57.413 INFO:tasks.workunit.client.0.smithi012.stdout:test api_lock_pp on pid 78586 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78586 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_lock_pp on pid 78586' 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78586 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_misc 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.415 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_misc' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_io.log 2024-03-26T04:47:57.416 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.417 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_misc 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.419 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.420 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.421 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.422 INFO:tasks.workunit.client.0.smithi012.stdout:test api_misc on pid 78601 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_misc 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78601 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_misc on pid 78601' 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78601 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78510/exe 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.429 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.430 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.431 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.431 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.431 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.431 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.431 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.433 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.434 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.435 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.436 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_misc_pp 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.437 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_misc_pp' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_misc_pp 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.438 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.439 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.440 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.441 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.442 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.443 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78526/exe 2024-03-26T04:47:57.444 INFO:tasks.workunit.client.0.smithi012.stdout:test api_misc_pp on pid 78627 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_misc_pp 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78627 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_misc_pp on pid 78627' 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78627 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.450 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.451 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_tier_pp 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_tier_pp' 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78544/exe 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-26T04:47:57.452 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.453 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.454 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_asio: /' 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_list.log 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78563/exe 2024-03-26T04:47:57.455 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_list: /' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.456 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.458 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_lock: /' 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.459 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.460 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.462 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_tier_pp 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.463 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.464 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.465 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.465 INFO:tasks.workunit.client.0.smithi012.stdout:test api_tier_pp on pid 78675 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_tier_pp 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78675 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_tier_pp on pid 78675' 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78675 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78601/exe 2024-03-26T04:47:57.470 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.471 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.472 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.473 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_misc: /' 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.474 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_pool 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_pool' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78586/exe 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.475 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.476 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_pool 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.477 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.478 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.479 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-26T04:47:57.480 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78627/exe 2024-03-26T04:47:57.480 INFO:tasks.workunit.client.0.smithi012.stdout:test api_pool on pid 78746 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_pool 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78746 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_pool on pid 78746' 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78746 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.488 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.489 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-26T04:47:57.490 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_snapshots 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_snapshots' 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.492 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_snapshots 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.493 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.494 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.495 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_snapshots 2024-03-26T04:47:57.499 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78805 2024-03-26T04:47:57.500 INFO:tasks.workunit.client.0.smithi012.stdout:test api_snapshots on pid 78805 2024-03-26T04:47:57.502 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_snapshots on pid 78805' 2024-03-26T04:47:57.502 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78805 2024-03-26T04:47:57.502 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.502 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_snapshots_pp 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_snapshots_pp' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_snapshots_pp 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.503 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.504 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78675/exe 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.505 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.506 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.506 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.506 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.506 INFO:tasks.workunit.client.0.smithi012.stdout:test api_snapshots_pp on pid 78816 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.508 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_snapshots_pp 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78816 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_snapshots_pp on pid 78816' 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78816 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_stat 2024-03-26T04:47:57.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_stat' 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_stat 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-26T04:47:57.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.511 INFO:tasks.workunit.client.0.smithi012.stdout:test api_stat on pid 78829 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_stat 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78829 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.513 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_stat on pid 78829' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78829 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.514 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.515 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_stat_pp 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_stat_pp' 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.516 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.517 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.518 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.519 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.520 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.521 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.521 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_stat_pp 2024-03-26T04:47:57.521 INFO:tasks.workunit.client.0.smithi012.stdout:test api_stat_pp on pid 78851 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_stat_pp 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78851 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_stat_pp on pid 78851' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78851 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_watch_notify 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_watch_notify' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.523 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_watch_notify 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.524 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.525 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.526 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.527 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.527 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.527 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.527 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.527 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.527 INFO:tasks.workunit.client.0.smithi012.stdout:test api_watch_notify on pid 78867 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_watch_notify 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78867 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_watch_notify on pid 78867' 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78867 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.531 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.532 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.533 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.534 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_watch_notify_pp 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_watch_notify_pp' 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.535 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.537 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_watch_notify_pp 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78805/exe 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78746/exe 2024-03-26T04:47:57.539 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.540 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.541 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.542 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_watch_notify_pp 2024-03-26T04:47:57.542 INFO:tasks.workunit.client.0.smithi012.stdout:test api_watch_notify_pp on pid 78899 2024-03-26T04:47:57.546 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.546 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.546 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78899 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_watch_notify_pp on pid 78899' 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78899 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.547 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_cmd 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.548 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.549 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_cmd' 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78816/exe 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_cmd 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_pool: /' 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.551 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.552 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.553 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.554 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.555 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.556 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.557 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78829/exe 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.559 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.560 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.560 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.560 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.560 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.560 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.560 INFO:tasks.workunit.client.0.smithi012.stdout:test api_cmd on pid 78946 2024-03-26T04:47:57.562 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_cmd 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78946 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_cmd on pid 78946' 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78946 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_cmd_pp 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.563 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_cmd_pp' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_stat: /' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.565 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_cmd_pp 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.566 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.567 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.568 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.569 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.570 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.571 INFO:tasks.workunit.client.0.smithi012.stdout:test api_cmd_pp on pid 78970 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_cmd_pp 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78970 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_cmd_pp on pid 78970' 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=78970 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.576 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.577 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_service 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_service' 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.578 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_service 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.579 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.580 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.581 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.582 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.583 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.584 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.585 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.585 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.585 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.585 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.585 INFO:tasks.workunit.client.0.smithi012.stdout:test api_service on pid 79006 2024-03-26T04:47:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_service 2024-03-26T04:47:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79006 2024-03-26T04:47:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_service on pid 79006' 2024-03-26T04:47:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79006 2024-03-26T04:47:57.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78899/exe 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.588 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.589 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_service_pp 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_service_pp' 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-26T04:47:57.590 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.591 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_service_pp 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78851/exe 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.592 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_service_pp 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79041 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.593 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-26T04:47:57.594 INFO:tasks.workunit.client.0.smithi012.stdout:test api_service_pp on pid 79041 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_service_pp on pid 79041' 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79041 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.597 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.598 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.599 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.600 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.601 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.602 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78867/exe 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.603 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78946/exe 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.604 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_c_write_operations 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_c_write_operations' 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.605 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-26T04:47:57.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_c_write_operations 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-26T04:47:57.607 INFO:tasks.workunit.client.0.smithi012.stdout:test api_c_write_operations on pid 79075 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_c_write_operations 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_cmd: /' 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79075 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_c_write_operations on pid 79075' 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79075 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_c_read_operations 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.612 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_c_read_operations' 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.614 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.615 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_c_read_operations 2024-03-26T04:47:57.616 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.617 INFO:tasks.workunit.client.0.smithi012.stdout:test api_c_read_operations on pid 79089 2024-03-26T04:47:57.618 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_c_read_operations 2024-03-26T04:47:57.618 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79089 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_c_read_operations on pid 79089' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79089 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s api_cls_remote_reads 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' api_cls_remote_reads' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.619 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo api_cls_remote_reads 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.620 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.621 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.622 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.623 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/78970/exe 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.624 INFO:tasks.workunit.client.0.smithi012.stdout:test api_cls_remote_reads on pid 79110 2024-03-26T04:47:57.626 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=api_cls_remote_reads 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79110 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test api_cls_remote_reads on pid 79110' 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79110 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s list_parallel 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.627 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' list_parallel' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.628 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.629 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.630 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.631 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo list_parallel 2024-03-26T04:47:57.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.633 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.634 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.635 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.636 INFO:tasks.workunit.client.0.smithi012.stdout:test list_parallel on pid 79151 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=list_parallel 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79151 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test list_parallel on pid 79151' 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79151 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.639 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.640 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.642 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s open_pools_parallel 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' open_pools_parallel' 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo open_pools_parallel 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.643 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.645 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.646 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79006/exe 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_service.log 2024-03-26T04:47:57.647 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.648 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.649 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.649 INFO:tasks.workunit.client.0.smithi012.stdout:test open_pools_parallel on pid 79205 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.652 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=open_pools_parallel 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79205 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test open_pools_parallel on pid 79205' 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79205 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:47:57.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_service: /' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s delete_pools_parallel 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' delete_pools_parallel' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo delete_pools_parallel 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.654 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.655 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.657 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=delete_pools_parallel 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79223 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test delete_pools_parallel on pid 79223' 2024-03-26T04:47:57.658 INFO:tasks.workunit.client.0.smithi012.stdout:test delete_pools_parallel on pid 79223 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79223 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s cls 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' cls' 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-26T04:47:57.661 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.662 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.663 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.664 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.665 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.666 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo cls 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.668 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.668 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.668 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.668 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.668 INFO:tasks.workunit.client.0.smithi012.stdout:test cls on pid 79237 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=cls 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79237 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test cls on pid 79237' 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79237 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-26T04:47:57.679 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.680 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.681 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.682 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.683 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.684 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.685 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.686 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79110/exe 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.687 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79151/exe 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.688 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.689 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.690 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.691 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.692 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ list_parallel: /' 2024-03-26T04:47:57.693 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79075/exe 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.694 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.695 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s cmd 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' cmd' 2024-03-26T04:47:57.696 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.697 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.698 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.699 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.700 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.701 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79089/exe 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.703 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.704 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.705 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79223/exe 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.706 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79205/exe 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.707 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo cmd 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-26T04:47:57.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-26T04:47:57.710 INFO:tasks.workunit.client.0.smithi012.stdout:test cmd on pid 79326 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=cmd 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79326 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test cmd on pid 79326' 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79326 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s handler_error 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' handler_error' 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.713 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo handler_error 2024-03-26T04:47:57.714 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.715 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79041/exe 2024-03-26T04:47:57.716 INFO:tasks.workunit.client.0.smithi012.stdout:test handler_error on pid 79352 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=handler_error 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79352 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test handler_error on pid 79352' 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79352 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s io 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' io' 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.720 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-26T04:47:57.721 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.722 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.723 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.724 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.725 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.726 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.727 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.728 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.729 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.729 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo io 2024-03-26T04:47:57.729 INFO:tasks.workunit.client.0.smithi012.stdout:test io on pid 79402 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=io 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79402 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test io on pid 79402' 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79402 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s ec_io 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.733 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.734 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79237/exe 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.735 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' ec_io' 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_cls 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_cls.log 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.736 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ cls: /' 2024-03-26T04:47:57.737 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo ec_io 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.738 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.740 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.741 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.742 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.742 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.742 INFO:tasks.workunit.client.0.smithi012.stdout:test ec_io on pid 79537 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=ec_io 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79537 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test ec_io on pid 79537' 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79537 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s list 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' list' 2024-03-26T04:47:57.746 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79326/exe 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.748 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_cmd 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo list 2024-03-26T04:47:57.749 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.750 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-26T04:47:57.751 INFO:tasks.workunit.client.0.smithi012.stdout:test list on pid 79561 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=list 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79561 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test list on pid 79561' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79561 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s ec_list 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' ec_list' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.754 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo ec_list 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ cmd: /' 2024-03-26T04:47:57.755 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.758 INFO:tasks.workunit.client.0.smithi012.stdout:test ec_list on pid 79573 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=ec_list 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79573 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test ec_list on pid 79573' 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79573 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s misc 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' misc' 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.762 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.763 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.764 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.765 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.766 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.768 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.769 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79352/exe 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79402/exe 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo misc 2024-03-26T04:47:57.770 INFO:tasks.workunit.client.0.smithi012.stdout:test misc on pid 79593 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.774 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=misc 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79593 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test misc on pid 79593' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79593 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_handler_error 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.775 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.776 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ handler_error: /' 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s pool 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' pool' 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ io: /' 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_io.log 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.778 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_io 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.781 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.782 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.783 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.785 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79561/exe 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo pool 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.786 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.787 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.788 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.789 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=pool 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79661 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test pool on pid 79661' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79661 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.790 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s read_operations 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' read_operations' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ list: /' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.791 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_list.log 2024-03-26T04:47:57.792 INFO:tasks.workunit.client.0.smithi012.stdout:test pool on pid 79661 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.796 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.797 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.798 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.799 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_list 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79573/exe 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo read_operations 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.800 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.801 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.802 INFO:tasks.workunit.client.0.smithi012.stdout:test read_operations on pid 79706 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=read_operations 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79706 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test read_operations on pid 79706' 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79706 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s snapshots 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' snapshots' 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.804 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.805 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo snapshots 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79593/exe 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.807 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.808 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.808 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_ec_list 2024-03-26T04:47:57.808 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-26T04:47:57.808 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.808 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=snapshots 2024-03-26T04:47:57.808 INFO:tasks.workunit.client.0.smithi012.stdout:test snapshots on pid 79718 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79718 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test snapshots on pid 79718' 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79718 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.814 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ ec_list: /' 2024-03-26T04:47:57.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_misc 2024-03-26T04:47:57.816 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ misc: /' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.817 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.818 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.819 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s watch_notify 2024-03-26T04:47:57.820 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' watch_notify' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_misc.log 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.821 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79537/exe 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.822 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.823 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.824 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.825 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo watch_notify 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_ec_io 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ ec_io: /' 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.826 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.827 INFO:tasks.workunit.client.0.smithi012.stdout:test watch_notify on pid 79767 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=watch_notify 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79767 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test watch_notify on pid 79767' 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79767 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++ printf %25s write_operations 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.832 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.833 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.834 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:+ r=' write_operations' 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo write_operations 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.835 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.836 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.838 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.839 INFO:tasks.workunit.client.0.smithi012.stdout:test write_operations on pid 79799 2024-03-26T04:47:57.846 INFO:tasks.workunit.client.0.smithi012.stderr:+ ff=write_operations 2024-03-26T04:47:57.846 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.846 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.846 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.846 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.846 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79799 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 'test write_operations on pid 79799' 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ pids[$f]=79799 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ ret=0 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79661/exe 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78675 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78675 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.847 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.848 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.849 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.850 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_pool.log 2024-03-26T04:47:57.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ pool: /' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_pool 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.852 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79718/exe 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.853 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79706/exe 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.854 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ snapshots: /' 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.855 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+ . /etc/bashrc 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ shopt -q login_shell 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_snapshots 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.856 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_read_operations 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ read_operations: /' 2024-03-26T04:47:57.857 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ umask 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++ SHELL=/bin/bash 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:47:57.858 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.859 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.860 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79767/exe 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.861 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_watch_notify 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:+++ return 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.862 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export S_COLORS=auto 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ S_COLORS=auto 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:47:57.863 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ watch_notify: /' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:47:57.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/locale 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:47:57.865 INFO:tasks.workunit.client.0.smithi012.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ unset LANG_backup config 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n '' ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:47:57.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '' ']' 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+++++ readlink /proc/79799/exe 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:++++ basename /usr/bin/bash 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_declare='declare -f' 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+++ which_opt=-f 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export which_declare 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+++ export -f which 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset i 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset -f pathmunge 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:47:57.867 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset rc 2024-03-26T04:47:57.868 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph_test_neorados_write_operations 2024-03-26T04:47:57.868 INFO:tasks.workunit.client.0.smithi012.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-26T04:47:57.868 INFO:tasks.workunit.client.0.smithi012.stderr:+ sed 's/^/ write_operations: /' 2024-03-26T04:47:58.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:57 smithi186 ceph-mon[33438]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:57 smithi012 ceph-mon[29863]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:58.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:57 smithi012 ceph-mon[41141]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:47:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:47:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:47:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:47:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:47:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:47:59.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:47:59.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:47:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:01.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1738465563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:01.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1621378002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:01.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[29863]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:48:01.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:01.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1738465563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:01.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1621378002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:01.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[41141]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:48:01.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:01.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1738465563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:01.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1621378002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:01.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:01 smithi186 ceph-mon[33438]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:48:01.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:02.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:48:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:48:05.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: pgmap v63: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:48:05.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:05.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2720644064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/695322663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1009290706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: pgmap v63: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2720644064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/695322663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1009290706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:05.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: pgmap v63: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:48:05.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:05.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2720644064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/695322663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1009290706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:05.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:48:05.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:05.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:05.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:06.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: pgmap v65: 324 pgs: 192 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3649843051' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1202235443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3396910549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/723142531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3927903188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi012-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:06.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3189600826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/898853451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2557651023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1613520692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2754707562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi012-78640-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1550705371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2948380165' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.215 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: pgmap v65: 324 pgs: 192 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3649843051' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1202235443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3396910549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/723142531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3927903188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi012-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:06.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3189600826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/898853451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2557651023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1613520692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2754707562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi012-78640-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1550705371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2948380165' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: pgmap v65: 324 pgs: 192 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3649843051' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1202235443' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3396910549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/723142531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3927903188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi012-78582-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi012-78552-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi012-79761-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:06.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3189600826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/898853451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2557651023' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1613520692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2754707562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi012-78640-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1550705371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2948380165' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:06.977 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:48:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:07.386 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-26T04:48:07.386 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [----------] Global test environment set-up. 2024-03-26T04:48:07.386 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (4 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (31 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (9 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-26T04:48:07.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (6 ms) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [----------] 12 tests from AsioRados (67 ms total) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [----------] Global test environment tear-down 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (9964 ms total) 2024-03-26T04:48:07.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-26T04:48:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: pgmap v67: 612 pgs: 288 unknown, 192 creating+peering, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: pgmap v68: 612 pgs: 288 unknown, 192 creating+peering, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-03-26T04:48:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/313063905' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T04:48:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3927903188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi012-78582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi012-78640-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1550705371' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2273810798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:07.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: pgmap v67: 612 pgs: 288 unknown, 192 creating+peering, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:07.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: pgmap v68: 612 pgs: 288 unknown, 192 creating+peering, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/313063905' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3927903188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi012-78582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi012-78640-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1550705371' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2273810798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: pgmap v67: 612 pgs: 288 unknown, 192 creating+peering, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: pgmap v68: 612 pgs: 288 unknown, 192 creating+peering, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-03-26T04:48:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/313063905' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T04:48:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3927903188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi012-78582-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi012-78644-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi012-78640-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1550705371' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi012-78567-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2273810798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:07.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.465 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:07.685 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-26T04:48:07.685 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] Global test environment set-up. 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.ListObjects (956 ms) 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (37 ms) 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo1 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo2 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo3 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T04:48:07.686 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo4 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo5 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo1 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset foo6,foo7 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo7 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo6 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns1:foo4 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns1:foo5 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns2:foo7 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns2:foo6 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns1:foo1 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: :foo1 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: :foo2 2024-03-26T04:48:07.687 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: :foo3 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (87 ms) 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 1 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 10 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 13 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 7 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 14 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 0 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 15 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 11 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 5 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 8 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 6 0 2024-03-26T04:48:07.688 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 3 0 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 4 0 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 12 0 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 9 0 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 2 0 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (69 ms) 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: x cursor=MIN 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=1 cursor=22:02547ec2:::1:head 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=10 cursor=22:52ea6a34:::10:head 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=13 cursor=22:566253c9:::13:head 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=7 cursor=22:5c6b0b28:::7:head 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=14 cursor=22:62a1935d:::14:head 2024-03-26T04:48:07.689 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=0 cursor=22:6cac518f:::0:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=15 cursor=22:863748b0:::15:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=11 cursor=22:89d3ae78:::11:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=5 cursor=22:b29083e3:::5:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=8 cursor=22:bd63b0f1:::8:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=6 cursor=22:c4fdafeb:::6:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=3 cursor=22:cfc208b3:::3:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=4 cursor=22:d83876eb:::4:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=12 cursor=22:de5d7c5f:::12:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=9 cursor=22:e960b815:::9:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > oid=2 cursor=22:f905c69b:::2:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=1 cursor=22:02547ec2:::1:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:02547ec2:::1:head 2024-03-26T04:48:07.690 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:02547ec2:::1:head -> 1 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=10 cursor=22:52ea6a34:::10:head 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:52ea6a34:::10:head 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:52ea6a34:::10:head -> 10 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=13 cursor=22:566253c9:::13:head 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:566253c9:::13:head 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:566253c9:::13:head -> 13 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=7 cursor=22:5c6b0b28:::7:head 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:5c6b0b28:::7:head 2024-03-26T04:48:07.691 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:5c6b0b28:::7:head -> 7 2024-03-26T04:48:07.753 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=14 cursor=22:62a1935d:::14:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:62a1935d:::14:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:62a1935d:::14:head -> 14 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=0 cursor=22:6cac518f:::0:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:6cac518f:::0:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:6cac518f:::0:head -> 0 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=15 cursor=22:863748b0:::15:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:863748b0:::15:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:863748b0:::15:head -> 15 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=11 cursor=22:89d3ae78:::11:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:89d3ae78:::11:head 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:89d3ae78:::11:head -> 11 2024-03-26T04:48:07.754 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=5 cursor=22:b29083e3:::5:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:b29083e3:::5:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:b29083e3:::5:head -> 5 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=8 cursor=22:bd63b0f1:::8:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:bd63b0f1:::8:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:bd63b0f1:::8:head -> 8 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=6 cursor=22:c4fdafeb:::6:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:c4fdafeb:::6:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:c4fdafeb:::6:head -> 6 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=3 cursor=22:cfc208b3:::3:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:cfc208b3:::3:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:cfc208b3:::3:head -> 3 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=4 cursor=22:d83876eb:::4:head 2024-03-26T04:48:07.755 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:d83876eb:::4:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:d83876eb:::4:head -> 4 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=12 cursor=22:de5d7c5f:::12:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:de5d7c5f:::12:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:de5d7c5f:::12:head -> 12 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=9 cursor=22:e960b815:::9:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:e960b815:::9:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:e960b815:::9:head -> 9 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : oid=2 cursor=22:f905c69b:::2:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:f905c69b:::2:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:f905c69b:::2:head -> 2 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:de5d7c5f:::12:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:de5d7c5f:::12:head expected=22:de5d7c5f:::12:head 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:de5d7c5f:::12:head -> 12 2024-03-26T04:48:07.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=12 expected=12 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:d83876eb:::4:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:d83876eb:::4:head expected=22:d83876eb:::4:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:d83876eb:::4:head -> 4 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=4 expected=4 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:cfc208b3:::3:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:cfc208b3:::3:head expected=22:cfc208b3:::3:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:cfc208b3:::3:head -> 3 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=3 expected=3 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:52ea6a34:::10:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:52ea6a34:::10:head expected=22:52ea6a34:::10:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:52ea6a34:::10:head -> 10 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=10 expected=10 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:02547ec2:::1:head 2024-03-26T04:48:07.757 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:02547ec2:::1:head expected=22:02547ec2:::1:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:02547ec2:::1:head -> 1 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=1 expected=1 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:c4fdafeb:::6:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:c4fdafeb:::6:head expected=22:c4fdafeb:::6:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:c4fdafeb:::6:head -> 6 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=6 expected=6 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:bd63b0f1:::8:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:bd63b0f1:::8:head expected=22:bd63b0f1:::8:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:bd63b0f1:::8:head -> 8 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=8 expected=8 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:b29083e3:::5:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:b29083e3:::5:head expected=22:b29083e3:::5:head 2024-03-26T04:48:07.758 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:b29083e3:::5:head -> 5 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=5 list_parallel: process_1_[79349]: starting. 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_1_[79349]: creating pool ceph_test_rados_list_parallel.smithi012-79261 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_1_[79349]: created object 0... 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_1_[79349]: created object 25... 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_1_[79349]: created object 49... 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_1_[79349]: finishing. 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_1_[79349]: shutting down. 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_2_[79350]: starting. 2024-03-26T04:48:07.919 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_2_[79350]: listing objects. 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_2_[79350]: listed object 0... 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_2_[79350]: listed object 25... 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_2_[79350]: saw 50 objects 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_2_[79350]: shutting down. 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: starting. 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: creating pool ceph_test_rados_list_parallel.smithi012-79261 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: created object 0... 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: created object 25... 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: created object 49... 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: finishing. 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_3_[80587]: shutting down. 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.920 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_4_[80588]: starting. 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_4_[80588]: listing objects. 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_4_[80588]: listed object 0... 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_4_[80588]: listed object 25... 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_4_[80588]: saw 43 objects 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_4_[80588]: shutting down. 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_5_[80589]: starting. 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_5_[80589]: removed 25 objects... 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_5_[80589]: removed half of the objects 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_5_[80589]: removed 50 objects... 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_5_[80589]: removed 50 objects 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_5_[80589]: shutting down. 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.921 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: starting. 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: creating pool ceph_test_rados_list_parallel.smithi012-79261 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: created object 0... 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: created object 25... 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: created object 49... 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: finishing. 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_6_[80659]: shutting down. 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: starting. 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: listing objects. 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: listed object 0... 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: listed object 25... 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: listed object 50... 2024-03-26T04:48:07.922 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: saw 56 objects 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_7_[80660]: shutting down. 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_8_[80661]: starting. 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_8_[80661]: added 25 objects... 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_8_[80661]: added half of the objects 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_8_[80661]: added 50 objects... 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_8_[80661]: added 50 objects 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_8_[80661]: shutting down. 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:07.923 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.118 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.118 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: starting. 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: creating pool ceph_test_rados_list_parallel.smithi012-79261 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: created object 0... 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: created object 25... 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: created object 49... 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: finishing. 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_9_[80707]: shutting down. 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: starting. 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: listing objects. 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: listed object 0... 2024-03-26T04:48:08.119 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: listed object 25... 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: listed object 50... 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: listed object 75... 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: saw 98 objects 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_10_[80708]: shutting down. 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_13_[80711]: starting. 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_13_[80711]: removed 25 objects... 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_13_[80711]: removed half of the objects 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_13_[80711]: removed 50 objects... 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_13_[80711]: removed 50 objects 2024-03-26T04:48:08.120 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_13_[80711]: shutting down. 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_11_[80709]: starting. 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_11_[80709]: added 25 objects... 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_11_[80709]: added half of the objects 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_11_[80709]: added 50 objects... 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_11_[80709]: added 50 objects 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_11_[80709]: shutting down. 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.121 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_12_[80710]: starting. 2024-03-26T04:48:08.122 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_12_[80710]: added 25 objects... 2024-03-26T04:48:08.122 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_12_[80710]: added half of the objects 2024-03-26T04:48:08.122 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_12_[80710]: added 50 objects... 2024-03-26T04:48:08.122 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_12_[80710]: added 50 objects 2024-03-26T04:48:08.122 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_12_[80710]: shutting down. 2024-03-26T04:48:08.122 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: starting. 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: creating pool ceph_test_rados_list_parallel.smithi012-79261 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: created object 0... 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: created object 25... 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: created object 49... 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: finishing. 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_14_[80789]: shutting down. 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:08.123 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: starting. 2024-03-26T04:48:08.124 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listing objects. 2024-03-26T04:48:08.124 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listed object 0... 2024-03-26T04:48:08.572 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listed object 2 handler_error: Running main() from gmock_main.cc 2024-03-26T04:48:08.572 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [----------] Global test environment set-up. 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (10834 ms) 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [----------] 1 test from neocls_handler_error (10834 ms total) 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [----------] Global test environment tear-down 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [==========] 1 test from 1 test suite ran. (10834 ms total) 2024-03-26T04:48:08.573 INFO:tasks.workunit.client.0.smithi012.stdout: handler_error: [ PASSED ] 1 test. 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: pgmap v71: 1188 pgs: 43 unknown, 852 creating+peering, 293 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3742171775' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3690205899' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:08.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: pgmap v71: 1188 pgs: 43 unknown, 852 creating+peering, 293 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:08.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3742171775' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:48:08.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:48:08.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:08.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3690205899' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:08.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:08.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:08.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: pgmap v71: 1188 pgs: 43 unknown, 852 creating+peering, 293 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi012-78916-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi012-79713-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi012-78725-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi012-78609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi012-78824-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi012-78928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79113-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi012-78949-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi012-79067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79256-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi012-79300-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi012-79038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3742171775' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3690205899' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:08.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1099936109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1099936109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1099936109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:48:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:10.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:10.482 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-26T04:48:10.482 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (328 ms) 2024-03-26T04:48:10.483 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-26T04:48:10.484 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (48 ms) 2024-03-26T04:48:10.484 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-26T04:48:10.484 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (9 ms) 2024-03-26T04:48:10.484 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-26T04:48:10.484 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (37 ms) 2024-03-26T04:48:10.484 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-26T04:48:10.485 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-03-26T04:48:10.485 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-26T04:48:10.485 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7 ms) 2024-03-26T04:48:10.485 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-26T04:48:10.485 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (7 ms) 2024-03-26T04:48:10.485 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-26T04:48:10.486 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2024-03-26T04:48:10.486 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-26T04:48:10.486 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (8 ms) 2024-03-26T04:48:10.486 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-26T04:48:10.486 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-03-26T04:48:10.486 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-26T04:48:10.487 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (7 ms) 2024-03-26T04:48:10.487 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-26T04:48:10.487 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (4 ms) 2024-03-26T04:48:10.487 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-26T04:48:10.487 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-03-26T04:48:10.487 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-26T04:48:10.488 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (11 ms) 2024-03-26T04:48:10.488 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-26T04:48:10.488 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-03-26T04:48:10.488 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-26T04:48:10.488 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-26T04:48:10.488 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (512 ms total) 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (12852 ms total) 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-26T04:48:10.489 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-26T04:48:10.490 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-26T04:48:10.490 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (12842 ms) 2024-03-26T04:48:10.490 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (12842 ms total) 2024-03-26T04:48:10.490 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: 2024-03-26T04:48:10.490 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-26T04:48:10.491 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (12842 ms total) 2024-03-26T04:48:10.491 INFO:tasks.workunit.client.0.smithi012.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-26T04:48:10.491 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-26T04:48:10.491 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:48:10.492 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-26T04:48:10.492 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-26T04:48:10.492 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-26T04:48:10.492 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (48 ms) 2024-03-26T04:48:10.493 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (70 ms) 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (12788 ms) 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (12906 ms total) 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-26T04:48:10.494 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (12906 ms total) 2024-03-26T04:48:10.495 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (8352 ms) 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (543 ms) 2024-03-26T04:48:11.012 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2962 ms) 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.266825+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi012-79761-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.266908+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.266973+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79070-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.273091+0000 mon.b [INF] from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.570555+0000 mon.b [INF] from='client.? 172.21.15.12:0/1099936109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.580467+0000 mon.a [INF] from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:08.632487+0000 mon.b [INF] from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.392662+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.392826+0000 mon.a [INF] from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.690179+0000 client.admin [INF] onexx 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.750199+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:11.013 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.750760+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:11.014 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.751184+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: pgmap v73: 1124 pgs: 166 unknown, 735 creating+peering, 223 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: onexx 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: pgmap v73: 1124 pgs: 166 unknown, 735 creating+peering, 223 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:48:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: onexx 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:11.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: pgmap v73: 1124 pgs: 166 unknown, 735 creating+peering, 223 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi012-79713-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: onexx 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:11.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: api5... 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listed object 50... 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listed object 75... 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listed object 100... 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: listed object 125... 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: saw 150 objects 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_15_[80790]: shutting down. 2024-03-26T04:48:11.554 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_16_[80791]: starting. 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_16_[80791]: added 25 objects... 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_16_[80791]: added half of the objects 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_16_[80791]: added 50 objects... 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_16_[80791]: added 50 objects 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: process_16_[80791]: shutting down. 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.555 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.556 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.556 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.556 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******************************* 2024-03-26T04:48:11.556 INFO:tasks.workunit.client.0.smithi012.stdout: list_parallel: ******* SUCCESS ********** 2024-03-26T04:48:12.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:48:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:12.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.2 deep-scrub starts 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.2 deep-scrub ok 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.5 deep-scrub starts 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.5 deep-scrub ok 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.8 deep-scrub starts 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.8 deep-scrub ok 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.3 deep-scrub starts 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.0 deep-scrub starts 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.3 deep-scrub ok 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: pgmap v76: 980 pgs: 303 unknown, 467 creating+peering, 210 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.0 deep-scrub ok 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.6 deep-scrub starts 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.6 deep-scrub ok 2024-03-26T04:48:12.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.7 deep-scrub starts 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.7 deep-scrub ok 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1847283530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.4 deep-scrub starts 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.4 deep-scrub ok 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3857649486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.9 deep-scrub starts 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: 11.9 deep-scrub ok 2024-03-26T04:48:12.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:12.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:12.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.2 deep-scrub starts 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.2 deep-scrub ok 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.5 deep-scrub starts 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.5 deep-scrub ok 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.8 deep-scrub starts 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.8 deep-scrub ok 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.3 deep-scrub starts 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.0 deep-scrub starts 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.3 deep-scrub ok 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: pgmap v76: 980 pgs: 303 unknown, 467 creating+peering, 210 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.0 deep-scrub ok 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.6 deep-scrub starts 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.6 deep-scrub ok 2024-03-26T04:48:12.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.7 deep-scrub starts 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.7 deep-scrub ok 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1847283530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.4 deep-scrub starts 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.4 deep-scrub ok 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3857649486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.9 deep-scrub starts 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: 11.9 deep-scrub ok 2024-03-26T04:48:12.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:12.912 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:12.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.2 deep-scrub starts 2024-03-26T04:48:12.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.2 deep-scrub ok 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.5 deep-scrub starts 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.5 deep-scrub ok 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.8 deep-scrub starts 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.8 deep-scrub ok 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.3 deep-scrub starts 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.0 deep-scrub starts 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.3 deep-scrub ok 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: pgmap v76: 980 pgs: 303 unknown, 467 creating+peering, 210 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.0 deep-scrub ok 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.6 deep-scrub starts 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.6 deep-scrub ok 2024-03-26T04:48:12.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.7 deep-scrub starts 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.7 deep-scrub ok 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1847283530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.4 deep-scrub starts 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.4 deep-scrub ok 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3857649486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.966 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.9 deep-scrub starts 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: 11.9 deep-scrub ok 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.967 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:12.968 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:12.968 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi012-78567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: 11.1 deep-scrub starts 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: 11.1 deep-scrub ok 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3857649486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:13.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: twoxx 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: pgmap v79: 812 pgs: 128 unknown, 237 creating+peering, 447 active+clean; 145 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 555 op/s 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:48:13.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi012-78567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:13.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:13.999 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: starting. 2024-03-26T04:48:13.999 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: creating pool ceph_test_rados_open_pools_parallel.smithi012-79319 2024-03-26T04:48:14.000 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: created object 0... 2024-03-26T04:48:14.000 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: created object 25... 2024-03-26T04:48:14.000 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: created object 49... 2024-03-26T04:48:14.000 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: finishing. 2024-03-26T04:48:14.000 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_1_[79369]: shutting down. 2024-03-26T04:48:14.000 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_2_[79370]: starting. 2024-03-26T04:48:14.001 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_2_[79370]: rados_pool_create. 2024-03-26T04:48:14.001 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_2_[79370]: rados_ioctx_create. 2024-03-26T04:48:14.001 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_2_[79370]: shutting down. 2024-03-26T04:48:14.001 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: ******************************* 2024-03-26T04:48:14.001 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: starting. 2024-03-26T04:48:14.002 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: creating pool ceph_test_rados_open_pools_parallel.smithi012-79319 2024-03-26T04:48:14.002 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: created object 0... 2024-03-26T04:48:14.002 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: created object 25... 2024-03-26T04:48:14.002 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: created object 49... 2024-03-26T04:48:14.002 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: finishing. 2024-03-26T04:48:14.002 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_3_[80861]: shutting down. 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: ******************************* 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_4_[80862]: starting. 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_4_[80862]: rados_pool_create. 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_4_[80862]: rados_ioctx_create. 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: process_4_[80862]: shutting down. 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: ******************************* 2024-03-26T04:48:14.003 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: ******************************* 2024-03-26T04:48:14.004 INFO:tasks.workunit.client.0.smithi012.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-26T04:48:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: 11.1 deep-scrub starts 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: 11.1 deep-scrub ok 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3857649486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: twoxx 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: pgmap v79: 812 pgs: 128 unknown, 237 creating+peering, 447 active+clean; 145 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 555 op/s 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi012-78567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: 11.1 deep-scrub starts 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: 11.1 deep-scrub ok 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3857649486' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi012-78552-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: twoxx 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: pgmap v79: 812 pgs: 128 unknown, 237 creating+peering, 447 active+clean; 145 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 555 op/s 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi012-78567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:14.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:14.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:14.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:14.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:14.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:14.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:15.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:15.056 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi012-78567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.057 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi012-78567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:15.058 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.059 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:15.060 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.060 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:15.060 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1188584678' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi012-78567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3795247299' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]: dispatch 2024-03-26T04:48:15.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:15.856 INFO:tasks.workunit.client.0.smithi012.stdout:_cmd: got: 2024-03-26T04:48:09.751602+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.751990+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.752379+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.752757+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.753163+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.753545+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:09.753936+0000 mon.c [INF] from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:10.466270+0000 mon.a [INF] from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:10.467393+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi012-79023-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.775021+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.775109+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:15.857 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.775183+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:15.858 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.775233+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:15.858 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.775283+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:15.858 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.775326+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:15.858 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.781682+0000 mon.c [INF] from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/193679371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/193679371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:48:16.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi012-78609-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi012-78582-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi012-78949-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi012-79761-1"}]': finished 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:16.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/193679371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:48:16.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:16.790 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.782277+0000 mon.b [INF] from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd pool create" api_io_pp: Running main() from gmock_main.cc 2024-03-26T04:48:16.790 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-26T04:48:16.790 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-26T04:48:16.790 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: seed 78609 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (277 ms) 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (19 ms) 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (23 ms) 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (7 ms) 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (28 ms) 2024-03-26T04:48:16.791 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (20 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (8 ms) 2024-03-26T04:48:16.792 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (4 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (9 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (16 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (5 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (456 ms total) 2024-03-26T04:48:16.793 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1968 ms) 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (12 ms) 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-26T04:48:16.794 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-03-26T04:48:17.154 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:48:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:48:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:17.209 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN cmd: Running main() from gmock_main.cc 2024-03-26T04:48:17.209 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:48:17.209 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [----------] Global test environment set-up. 2024-03-26T04:48:17.209 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9289 ms) 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (5625 ms) 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (4580 ms) 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [----------] 3 tests from NeoRadosCmd (19494 ms total) 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [----------] Global test environment tear-down 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [==========] 3 tests from 1 test suite ran. (19494 ms total) 2024-03-26T04:48:17.210 INFO:tasks.workunit.client.0.smithi012.stdout: cmd: [ PASSED ] 3 tests. 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout:, "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.782708+0000 mon.b [INF] from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.784688+0000 mon.b [INF] from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.831265+0000 mon.b [INF] from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:14.922298+0000 mon.c [INF] from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.072754+0000 mon.b [INF] from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.072799+0000 mon.c [INF] from='client.? 172.21.15.12:0/193679371' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.075712+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.706480+0000 client.admin [INF] threexx 2024-03-26T04:48:17.213 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.858726+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.214 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.859110+0000 mon.a [INF] from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:48:17.214 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.859270+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:17.214 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.859599+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:17.214 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.859878+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: pgmap v82: 820 pgs: 328 unknown, 114 creating+peering, 378 active+clean; 145 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 546 op/s 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: threexx 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2684212471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:17.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:17.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: pgmap v82: 820 pgs: 328 unknown, 114 creating+peering, 378 active+clean; 145 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 546 op/s 2024-03-26T04:48:17.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: threexx 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2684212471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: pgmap v82: 820 pgs: 328 unknown, 114 creating+peering, 378 active+clean; 145 MiB data, 726 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 546 op/s 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: threexx 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:17.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2684212471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:17.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:18.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: pgmap v85: 924 pgs: 153 creating+peering, 25 unknown, 746 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 14 op/s 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2"}]: dispatch 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:18.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: pgmap v85: 924 pgs: 153 creating+peering, 25 unknown, 746 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 14 op/s 2024-03-26T04:48:18.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2"}]: dispatch 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:48:18.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T04:48:18.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:18.659 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:48:18.659 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:48:18.659 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.659 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:18.659 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:18.659 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:48:18.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: pgmap v85: 924 pgs: 153 creating+peering, 25 unknown, 746 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s wr, 14 op/s 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi012-78644-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:18.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2"}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:18.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi012-78582-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi012-78949-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:48:18.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:48:18.756 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:15.860175+0000 mon.a [INF] from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi012-78725-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi012-78725-10"}]': fi delete_pools_parallel: process_1_[79360]: starting. 2024-03-26T04:48:18.756 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_1_[79360]: creating pool ceph_test_rados_delete_pools_parallel.smithi012-79309 2024-03-26T04:48:18.756 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_1_[79360]: created object 0... 2024-03-26T04:48:18.756 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_1_[79360]: created object 25... 2024-03-26T04:48:18.756 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_1_[79360]: created object 49... 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_1_[79360]: finishing. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_1_[79360]: shutting down. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_2_[79361]: starting. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_2_[79361]: deleting pool ceph_test_rados_delete_pools_parallel.smithi012-79309 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_2_[79361]: shutting down. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: starting. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: creating pool ceph_test_rados_delete_pools_parallel.smithi012-79309 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: created object 0... 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: created object 25... 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: created object 49... 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: finishing. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_3_[80855]: shutting down. 2024-03-26T04:48:18.757 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_5_[80857]: starting. 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_5_[80857]: listing objects. 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_5_[80857]: listed object 0... 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_5_[80857]: listed object 25... 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_5_[80857]: saw 50 objects 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_5_[80857]: shutting down. 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_4_[80856]: starting. 2024-03-26T04:48:18.758 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_4_[80856]: deleting pool ceph_test_rados_delete_pools_parallel.smithi012-79309 2024-03-26T04:48:18.759 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: process_4_[80856]: shutting down. 2024-03-26T04:48:18.759 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:48:18.759 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:48:18.759 INFO:tasks.workunit.client.0.smithi012.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: Running main() from gmock_main.cc 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [----------] Global test environment set-up. 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [ OK ] NeoRadosCls.DNE (11708 ms) 2024-03-26T04:48:18.762 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-26T04:48:18.763 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (9355 ms) 2024-03-26T04:48:18.763 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [----------] 2 tests from NeoRadosCls (21063 ms total) 2024-03-26T04:48:18.763 INFO:tasks.workunit.client.0.smithi012.stdout: cls: 2024-03-26T04:48:18.763 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [----------] Global test environment tear-down 2024-03-26T04:48:18.763 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [==========] 2 tests from 1 test suite ran. (21063 ms total) 2024-03-26T04:48:18.763 INFO:tasks.workunit.client.0.smithi012.stdout: cls: [ PASSED ] 2 tests. 2024-03-26T04:48:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: fourxx 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:19.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: fourxx 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:19.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:19.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:48:19.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:19.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:19.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:48:19.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:48:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: fourxx 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3890312490' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2336553698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:19.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:19.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:48:19.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-26T04:48:19.852 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2055 ms total) 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: 2024-03-26T04:48:19.853 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-26T04:48:19.854 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (22465 ms total) 2024-03-26T04:48:19.854 INFO:tasks.workunit.client.0.smithi012.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: handle_notify cookie 94021178272240 notify_id 335007449088 notifier_gid 24808 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (14027 ms) 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-26T04:48:20.856 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: Trying... 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: handle_notify cookie 94021190378576 notify_id 365072220160 notifier_gid 25129 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: Waiting for 3.000000000s 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: Timed out. 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: Flushing... 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: Flushed... 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (9009 ms) 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (23036 ms total) 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (23036 ms total) 2024-03-26T04:48:20.857 INFO:tasks.workunit.client.0.smithi012.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-26T04:48:20.868 INFO:tasks.workunit.client.0.smithi012.stdout: list: Running main() from gmock_main.cc 2024-03-26T04:48:20.868 INFO:tasks.workunit.client.0.smithi012.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:48:20.868 INFO:tasks.workunit.client.0.smithi012.stdout: list: [----------] Global test environment set-up. 2024-03-26T04:48:20.868 INFO:tasks.workunit.client.0.smithi012.stdout: list: [----------] 3 tests from NeoradosList 2024-03-26T04:48:20.868 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-26T04:48:20.868 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ OK ] NeoradosList.ListObjects (10815 ms) 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ OK ] NeoradosList.ListObjectsNS (5425 ms) 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ OK ] NeoradosList.ListObjectsMany (6858 ms) 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [----------] 3 tests from NeoradosList (23098 ms total) 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [----------] Global test environment tear-down 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [==========] 3 tests from 1 test suite ran. (23098 ms total) 2024-03-26T04:48:20.869 INFO:tasks.workunit.client.0.smithi012.stdout: list: [ PASSED ] 3 tests. 2024-03-26T04:48:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: pgmap v88: 876 pgs: 76 creating+peering, 171 unknown, 629 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-03-26T04:48:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1809194654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/896075632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:20.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:20.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi012-79713-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: pgmap v88: 876 pgs: 76 creating+peering, 171 unknown, 629 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1809194654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/896075632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:20.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:20.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:48:20.910 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:48:20.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.911 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi012-79713-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: pgmap v88: 876 pgs: 76 creating+peering, 171 unknown, 629 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1809194654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/896075632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3211443372' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T04:48:20.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi012-79761-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi012-78644-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi012-79713-1"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi012-78609-22"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-4", "overlaypool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:48:20.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1195208943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1318687444' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]: dispatch 2024-03-26T04:48:20.965 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi012-79713-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:21.329 INFO:tasks.workunit.client.0.smithi012.stdout:nished 2024-03-26T04:48:21.329 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.141438+0000 mon.b [INF] from='client.? 172.21.15.12:0/2684212471' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi012-78552-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:21.329 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.142540+0000 mon.a [INF] from='client.? 172.21.15.12:0/2657578513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:21.329 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.143854+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi012-79761-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:21.329 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.146894+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi012-78688-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.147837+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.653337+0000 mon.a [WRN] Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.659851+0000 mon.a [INF] from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: got: 2024-03-26T04:48:16.662171+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi012-79038-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (11903 ms) 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (23760 ms total) 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (23760 ms total) 2024-03-26T04:48:21.330 INFO:tasks.workunit.client.0.smithi012.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: seed 79038 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (372 ms) 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (1038 ms) 2024-03-26T04:48:21.947 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (8 ms) 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (12 ms) 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (1431 ms total) 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-26T04:48:21.948 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1180 ms) 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1011 ms) 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (12 ms) 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2204 ms total) 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (24398 ms total) 2024-03-26T04:48:21.949 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-26T04:48:22.212 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: pgmap v91: 780 pgs: 76 creating+peering, 211 unknown, 493 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]': finished 2024-03-26T04:48:22.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi012-79713-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi012-79713-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]: dispatch 2024-03-26T04:48:22.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:22.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: pgmap v91: 780 pgs: 76 creating+peering, 211 unknown, 493 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]': finished 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi012-79713-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:22.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi012-79713-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:22.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: pgmap v91: 780 pgs: 76 creating+peering, 211 unknown, 493 active+clean; 145 MiB data, 765 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi012-78688-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-4-cache", "mode": "writeback"}]': finished 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi012-79023-1"}]': finished 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi012-79713-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:22.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi012-79713-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1420327435' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]: dispatch 2024-03-26T04:48:22.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:23.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:23.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: pool 'PoolQuotaPP_smithi012-78567-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T04:48:23.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:48:23.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:23.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:23.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2784479939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2055988395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:23.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]: dispatch 2024-03-26T04:48:23.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:23.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: pool 'PoolQuotaPP_smithi012-78567-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T04:48:23.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2784479939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2055988395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]: dispatch 2024-03-26T04:48:23.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: pool 'PoolQuotaPP_smithi012-78567-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi012-79038-7"}]': finished 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/260282239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/67716164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2784479939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2055988395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:23.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]: dispatch 2024-03-26T04:48:24.156 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: Running main() from gmock_main.cc 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [----------] Global test environment set-up. 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (884 ms) 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: no timeout :/ 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: no timeout :/ 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: no timeout :/ 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: no timeout :/ 2024-03-26T04:48:24.157 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: no timeout :/ 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (70 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (10 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (6 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (7 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-26T04:48:24.158 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.TruncTest (6 ms) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.RmXattr (15 ms) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIo.XattrIter (7 ms) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [----------] 14 tests from LibRadosIo (1024 ms total) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1110 ms) 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-26T04:48:24.159 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (11 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (6 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-03-26T04:48:24.160 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-26T04:48:24.161 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-26T04:48:24.161 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1161 ms total) 2024-03-26T04:48:24.161 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [---- api_stat: Running main() from gmock_main.cc 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [----------] Global test environment set-up. 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStat.Stat (323 ms) 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-26T04:48:24.163 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (57 ms) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStat.StatNS (20 ms) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [----------] 5 tests from LibRadosStat (404 ms total) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1101 ms) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (20 ms) 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-26T04:48:24.164 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1124 ms total) 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [----------] Global test environment tear-down 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (26663 ms total) 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-26T04:48:24.165 INFO:tasks.workunit.client.0.smithi012.stdout:------] Global test environment tear-down 2024-03-26T04:48:24.166 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [==========] 24 tests from 2 test suites ran. (26782 ms total) 2024-03-26T04:48:24.166 INFO:tasks.workunit.client.0.smithi012.stdout: api_io: [ PASSED ] 24 tests. 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: pgmap v94: 756 pgs: 96 creating+peering, 96 unknown, 564 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 2 op/s 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi012-79713-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]': finished 2024-03-26T04:48:24.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:24.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: pgmap v94: 756 pgs: 96 creating+peering, 96 unknown, 564 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 2 op/s 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi012-79713-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:24.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]': finished 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:24.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:24.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: pgmap v94: 756 pgs: 96 creating+peering, 96 unknown, 564 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 2 op/s 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi012-79713-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi012-79067-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-4"}]': finished 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2838625226' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:48:24.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:24.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:24.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]: dispatch 2024-03-26T04:48:24.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.324 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: seed 78725 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (328 ms) 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (46 ms) 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1090 ms) 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1020 ms) 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-26T04:48:25.325 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (49 ms) 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (23 ms) 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2563 ms total) 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (2727 ms) 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-26T04:48:25.326 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (9 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1067 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1083 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (10 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (6 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (19 ms) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (4925 ms total) 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: 2024-03-26T04:48:25.327 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (27887 ms total) 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: Running main() from gmock_main.cc 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [----------] Global test environment set-up. 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (17002 ms) 2024-03-26T04:48:25.328 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-26T04:48:25.329 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (10538 ms) 2024-03-26T04:48:25.329 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (27540 ms total) 2024-03-26T04:48:25.329 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: 2024-03-26T04:48:25.329 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [----------] Global test environment tear-down 2024-03-26T04:48:25.329 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (27540 ms total) 2024-03-26T04:48:25.329 INFO:tasks.workunit.client.0.smithi012.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-26T04:48:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2795541804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]: dispatch 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2795541804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]: dispatch 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:25.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:25.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:25.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi012-78949-7"}]': finished 2024-03-26T04:48:25.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi012-78582-16"}]': finished 2024-03-26T04:48:25.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi012-78552-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-4", "tierpool": "test-rados-api-smithi012-78824-4-cache"}]': finished 2024-03-26T04:48:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2795541804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:25.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2853649765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.718 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]: dispatch 2024-03-26T04:48:25.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:48:25.719 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]: dispatch 2024-03-26T04:48:25.720 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:25.720 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]: dispatch 2024-03-26T04:48:25.720 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:26.542 INFO:tasks.workunit.client.0.smithi012.stdout: pool: Running main() from gmock_main.cc 2024-03-26T04:48:26.542 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-26T04:48:26.542 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [----------] Global test environment set-up. 2024-03-26T04:48:26.542 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-26T04:48:26.542 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ OK ] NeoRadosPools.PoolList (9535 ms) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (6667 ms) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3207 ms) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3645 ms) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (3300 ms) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2379 ms) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [----------] 6 tests from NeoRadosPools (28733 ms total) 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [----------] Global test environment tear-down 2024-03-26T04:48:26.543 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [==========] 6 tests from 1 test suite ran. (28733 ms total) 2024-03-26T04:48:26.544 INFO:tasks.workunit.client.0.smithi012.stdout: pool: [ PASSED ] 6 tests. 2024-03-26T04:48:26.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: pgmap v96: 700 pgs: 64 creating+peering, 144 unknown, 492 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2024-03-26T04:48:26.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:26.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:26.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:26.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:48:26.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:26.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:26.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:26.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:26.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:26.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: pgmap v96: 700 pgs: 64 creating+peering, 144 unknown, 492 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:26.906 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:48:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:48:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:26.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: pgmap v96: 700 pgs: 64 creating+peering, 144 unknown, 492 active+clean; 145 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2024-03-26T04:48:26.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4047816427' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi012-78725-10"}]': finished 2024-03-26T04:48:26.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi012-79761-2"}]': finished 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:26.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [----------] Global test environment set-up. 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5066 ms) 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ OK ] LibRadosService.RegisterLate (28 ms) 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: cluster: 2024-03-26T04:48:27.387 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: id: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: health: HEALTH_WARN 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2 pool(s) do not have an application enabled 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: services: 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: data: 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: pools: 8 pools, 196 pgs 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T04:48:27.388 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: pgs: 32.653% pgs unknown 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 132 active+clean 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 64 unknown 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: io: 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: cluster: 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: id: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: health: HEALTH_WARN 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 25 pool(s) do not have an application enabled 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.389 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: services: 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: data: 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: pools: 39 pools, 1188 pgs 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: objects: 231 objects, 582 KiB 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: usage: 236 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:27.390 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: pgs: 3.620% pgs unknown 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 71.717% pgs not active 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 852 creating+peering 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 293 active+clean 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 43 unknown 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: io: 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: client: 1.7 KiB/s rd, 0 B/s wr, 1 op/s rd, 0 op/s wr 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4561 ms) 2024-03-26T04:48:27.391 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-26T04:48:27.392 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ OK ] LibRadosService.Status (20101 ms) 2024-03-26T04:48:27.392 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [----------] 4 tests from LibRadosService (29756 ms total) 2024-03-26T04:48:27.392 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: 2024-03-26T04:48:27.392 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [----------] Global test environment tear-down 2024-03-26T04:48:27.392 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29756 ms total) 2024-03-26T04:48:27.392 INFO:tasks.workunit.client.0.smithi012.stdout: api_service: [ PASSED ] 4 tests. 2024-03-26T04:48:27.759 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (380 ms) 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94212291979216 err -107 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (21 ms) 2024-03-26T04:48:27.760 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94212291979216 err -107 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1021 ms) 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 335007449088 cookie 94212291979216 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (14 ms) 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 335007449089 cookie 94212292012304 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 335007449090 cookie 94212292012304 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5 ms) 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-26T04:48:27.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 335007449089 cookie 94212292051824 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 335007449089 cookie 94212292073696 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (7 ms) 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 335007449091 cookie 94212292073696 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 339302416388 cookie 94212292073696 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3004 ms) 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94212292073696 err -107 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_cb from 24676 notify_id 365072220162 cookie 94212292054000 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5008 ms) 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-26T04:48:27.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94212292073696 err -107 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10471 ms total) 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1358 ms) 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1358 ms total) 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (30201 ms total) 2024-03-26T04:48:27.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-26T04:48:27.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]: dispatch 2024-03-26T04:48:27.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: pgmap v99: 644 pgs: 96 creating+peering, 548 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]': finished 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4189714844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:28.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]: dispatch 2024-03-26T04:48:28.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: pgmap v99: 644 pgs: 96 creating+peering, 548 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]': finished 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4189714844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:28.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:28.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:28.165 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: pgmap v99: 644 pgs: 96 creating+peering, 548 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-79023-6", "pg_num": 4}]': finished 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2481663038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4189714844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi012-79713-2"}]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:28.166 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:28.167 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]: dispatch 2024-03-26T04:48:28.167 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.167 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:28.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: Running main() from gmock_main.cc 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [----------] Global test environment set-up. 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (71 ms) 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.LockShared (1225 ms) 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1155 ms) 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1116 ms) 2024-03-26T04:48:28.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (33 ms) 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.Unlock (23 ms) 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [----------] 8 tests from LibRadosLock (3632 ms total) 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1144 ms) 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-26T04:48:28.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (45 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1065 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1109 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (6 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (26 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (21 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (10 ms) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3426 ms total) 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: 2024-03-26T04:48:28.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [----------] Global test environment tear-down 2024-03-26T04:48:28.766 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (31360 ms total) 2024-03-26T04:48:28.766 INFO:tasks.workunit.client.0.smithi012.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-26T04:48:28.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3120573593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3769781502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:28.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:28.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:29.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:48:29.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:29.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:29.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3120573593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3769781502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:29.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2872637419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi012-79713-2"}]': finished 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi012-79067-12"}]': finished 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi012-78916-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1949473702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3120573593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3769781502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:29.158 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: pgmap v102: 616 pgs: 164 unknown, 64 creating+peering, 388 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]': finished 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:31.160 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]: dispatch 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]: dispatch 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4098000435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: pgmap v102: 616 pgs: 164 unknown, 64 creating+peering, 388 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]': finished 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]: dispatch 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]: dispatch 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4098000435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:31.162 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:31.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: pgmap v102: 616 pgs: 164 unknown, 64 creating+peering, 388 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:48:31.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi012-78644-10"}]': finished 2024-03-26T04:48:31.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "overlaypool": "test-rados-api-smithi012-79023-6"}]': finished 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi012-78552-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi012-79713-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]: dispatch 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]: dispatch 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4098000435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:31.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5044 ms) 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (2420 ms) 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-26T04:48:31.623 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (21036 ms) 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: attempt 0 of 20 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: failed to deregister: 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: { 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "epoch": 20, 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "modified": "2024-03-26T04:48:28.129490+0000", 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "services": { 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "laundry": { 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "daemons": { 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "summary": "", 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "close-test-pid79356": { 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_epoch": 20, 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_stamp": "2024-03-26T04:48:26.460492+0000", 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "gid": 18840, 2024-03-26T04:48:31.624 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "addr": "172.21.15.12:0/2331464541", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "metadata": { 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro": "centos", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_version": "9", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "foo": "bar", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "hostname": "smithi012", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T04:48:31.625 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_total_kb": "32494636", 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "os": "Linux", 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "this": "that" 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "task_status": {} 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "pid79356": { 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_epoch": 19, 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_stamp": "2024-03-26T04:48:25.195090+0000", 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "gid": 24916, 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "addr": "172.21.15.12:0/1319574653", 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "metadata": { 2024-03-26T04:48:31.626 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro": "centos", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_version": "9", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "foo": "bar", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "hostname": "smithi012", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_total_kb": "32494636", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "os": "Linux", 2024-03-26T04:48:31.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "this": "that" 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "task_status": {} 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "rgw": { 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "daemons": { 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "summary": "", 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "14436": { 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_epoch": 12, 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_stamp": "2024-03-26T04:45:27.898870+0000", 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "gid": 14436, 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "addr": "172.21.15.12:0/4280417874", 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "metadata": { 2024-03-26T04:48:31.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "container_hostname": "smithi012", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro": "centos", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_version": "8", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "hostname": "smithi012", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "id": "foo.a", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T04:48:31.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_total_kb": "32494636", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "num_handles": "1", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "os": "Linux", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "pid": "2", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "realm_id": "", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "realm_name": "", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zone_id": "f87c8106-6d08-4ff5-be50-bc7389e33d34", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zone_name": "default", 2024-03-26T04:48:31.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zonegroup_id": "c27bb0de-7d96-4ff7-880a-a7fd7414e1ee", 2024-03-26T04:48:32.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:48:31 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:48:32.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:32.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:32.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]': finished 2024-03-26T04:48:32.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: pgmap v105: 520 pgs: 196 unknown, 32 creating+peering, 292 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1894336863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3552542576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/100885173' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:32.656 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]': finished 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: pgmap v105: 520 pgs: 196 unknown, 32 creating+peering, 292 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1894336863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3552542576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/100885173' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-79023-6", "mode": "writeback"}]': finished 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: pgmap v105: 520 pgs: 196 unknown, 32 creating+peering, 292 active+clean; 145 MiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1894336863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3552542576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi012-79713-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/100885173' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zonegroup_name": "def api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2289 ms) 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (6 ms) 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2295 ms total) 2024-03-26T04:48:33.422 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (224 ms) 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5829 ms) 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (11 ms) 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (15 ms) 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442544928 notify_id 420906795012 notifier_gid 25387 2024-03-26T04:48:33.423 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (23 ms) 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442530800 notify_id 420906795013 notifier_gid 25387 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (25 ms) 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442544928 notify_id 420906795014 notifier_gid 25387 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (12 ms) 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442530800 notify_id 420906795015 notifier_gid 25387 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (36 ms) 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442544928 notify_id 420906795016 notifier_gid 25387 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (14 ms) 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-26T04:48:33.424 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442530800 notify_id 420906795017 notifier_gid 25387 2024-03-26T04:48:33.425 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (44 ms) 2024-03-26T04:48:33.425 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-26T04:48:33.425 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: trying... 2024-03-26T04:48:33.425 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442530800 notify_id 420906795011 notifier_gid 25387 2024-03-26T04:48:33.425 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: timed out 2024-03-26T04:48:33.425 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: flushing 2024-03-26T04:48:33.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: pgmap v107: 592 pgs: 72 unknown, 96 creating+peering, 424 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/532103590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/60598780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: pgmap v107: 592 pgs: 72 unknown, 96 creating+peering, 424 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/532103590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/60598780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:33.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:33.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.658 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: pgmap v107: 592 pgs: 72 unknown, 96 creating+peering, 424 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:48:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/532103590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/60598780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:33.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:34.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:48:34.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:34.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:34.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:48:34.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:34.909 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:34.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi012-78928-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi012-78552-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-79274-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:34.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: pgmap v110: 624 pgs: 232 unknown, 64 creating+peering, 328 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-26T04:48:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:48:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:35.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:35 smithi186 ceph-mon[33438]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:48:36.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: pgmap v110: 624 pgs: 232 unknown, 64 creating+peering, 328 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-26T04:48:36.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:48:36.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:36.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:36.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:36.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:36.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[29863]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:48:36.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: pgmap v110: 624 pgs: 232 unknown, 64 creating+peering, 328 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-26T04:48:36.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:48:36.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:36.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:36.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:36.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:36.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:35 smithi012 ceph-mon[41141]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:48:36.727 INFO:tasks.workunit.client.0.smithi012.stdout: ault" 2024-03-26T04:48:36.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:36.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "task_status": {} 2024-03-26T04:48:36.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:36.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:36.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: attempt 1 of 20 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: failed to deregister: 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: { 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "epoch": 22, 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "modified": "2024-03-26T04:48:32.132832+0000", 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "services": { 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "laundry": { 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "daemons": { 2024-03-26T04:48:36.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "summary": "", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "close-test-pid79356": { 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_epoch": 22, 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_stamp": "2024-03-26T04:48:31.670674+0000", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "gid": 34357, 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "addr": "172.21.15.12:0/3053756019", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "metadata": { 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-26T04:48:36.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro": "centos", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_version": "9", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "foo": "bar", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "hostname": "smithi012", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_total_kb": "32494636", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "os": "Linux", 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "this": "that" 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:36.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "task_status": {} 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "pid79356": { 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_epoch": 19, 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_stamp": "2024-03-26T04:48:25.195090+0000", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "gid": 24916, 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "addr": "172.21.15.12:0/1319574653", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "metadata": { 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-26T04:48:36.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro": "centos", 2024-03-26T04:48:36.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T04:48:36.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_version": "9", 2024-03-26T04:48:36.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "foo": "bar", 2024-03-26T04:48:36.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "hostname": "smithi012", 2024-03-26T04:48:37.154 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:48:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:48:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4286224361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: pgmap v113: 592 pgs: 64 creating+peering, 528 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:48:37.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:37.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/86734878' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:37.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:37 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4286224361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: pgmap v113: 592 pgs: 64 creating+peering, 528 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/86734878' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4286224361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:37.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:37.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:37.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: pgmap v113: 592 pgs: 64 creating+peering, 528 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:48:37.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:37.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/86734878' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:37.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:37 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:38.886 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:38.886 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:48:38.886 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1473608912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.886 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4058092562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.887 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:38.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:48:38.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1473608912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:38.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4058092562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1473608912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi012-78567-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:38.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:38.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:48:38.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:38.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:38.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:38.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4058092562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:38.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: api_serv ec_list: Running main() from gmock_main.cc 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [----------] Global test environment set-up. 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (22076 ms) 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7897 ms) 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (12571 ms) 2024-03-26T04:48:40.324 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [----------] 3 tests from NeoradosECList (42544 ms total) 2024-03-26T04:48:40.325 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: 2024-03-26T04:48:40.325 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [----------] Global test environment tear-down 2024-03-26T04:48:40.325 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (42544 ms total) 2024-03-26T04:48:40.325 INFO:tasks.workunit.client.0.smithi012.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-26T04:48:40.392 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: pgmap v116: 712 pgs: 160 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]': finished 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: pgmap v116: 712 pgs: 160 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:40.393 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]': finished 2024-03-26T04:48:40.394 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:40.394 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:40.394 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:40.394 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:48:40.394 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:40.394 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: pgmap v116: 712 pgs: 160 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:48:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-7"}]': finished 2024-03-26T04:48:40.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:40.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi012-78552-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:40.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2079616535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:40.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:48:40.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]: dispatch 2024-03-26T04:48:40.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:41.376 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-26T04:48:41.376 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-26T04:48:41.376 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-26T04:48:41.377 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-26T04:48:41.377 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-26T04:48:41.377 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-26T04:48:41.377 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-26T04:48:41.377 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: 2024-03-26T04:48:41.377 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-26T04:48:41.378 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-26T04:48:41.378 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (10965 ms) 2024-03-26T04:48:41.378 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-26T04:48:41.378 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (5685 ms) 2024-03-26T04:48:41.378 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-26T04:48:41.378 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3497 ms) 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3117 ms) 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4440 ms) 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (5207 ms) 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (10849 ms) 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (43760 ms total) 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-26T04:48:41.379 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (43760 ms total) 2024-03-26T04:48:41.380 INFO:tasks.workunit.client.0.smithi012.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-26T04:48:41.559 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:41.559 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:41.559 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:41.559 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:41.559 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: pgmap v118: 616 pgs: 64 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:41.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1602716951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi012-78567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: 11.2 deep-scrub starts 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: 11.2 deep-scrub ok 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: 11.5 deep-scrub starts 2024-03-26T04:48:41.561 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[29863]: 11.5 deep-scrub ok 2024-03-26T04:48:41.562 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:41.562 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:41.562 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:41.562 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:41.562 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: pgmap v118: 616 pgs: 64 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:41.563 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T04:48:41.564 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1602716951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi012-78567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:41.565 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:41.565 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:41.565 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: 11.2 deep-scrub starts 2024-03-26T04:48:41.565 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: 11.2 deep-scrub ok 2024-03-26T04:48:41.565 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: 11.5 deep-scrub starts 2024-03-26T04:48:41.565 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:41 smithi012 ceph-mon[41141]: 11.5 deep-scrub ok 2024-03-26T04:48:41.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: pgmap v118: 616 pgs: 64 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-03-26T04:48:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi012-79713-3"}]': finished 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1602716951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi012-78567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:41.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:41.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: 11.2 deep-scrub starts 2024-03-26T04:48:41.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: 11.2 deep-scrub ok 2024-03-26T04:48:41.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: 11.5 deep-scrub starts 2024-03-26T04:48:41.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:41 smithi186 ceph-mon[33438]: 11.5 deep-scrub ok 2024-03-26T04:48:42.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:48:42.520 INFO:tasks.workunit.client.0.smithi012.stdout:ice_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T04:48:42.520 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T04:48:42.520 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T04:48:42.520 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_total_kb": "32494636", 2024-03-26T04:48:42.520 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "os": "Linux", 2024-03-26T04:48:42.520 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "this": "that" 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "task_status": {} 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "rgw": { 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "daemons": { 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "summary": "", 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "14436": { 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_epoch": 12, 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "start_stamp": "2024-03-26T04:45:27.898870+0000", 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "gid": 14436, 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "addr": "172.21.15.12:0/4280417874", 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "metadata": { 2024-03-26T04:48:42.521 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "container_hostname": "smithi012", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro": "centos", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "distro_version": "8", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "hostname": "smithi012", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "id": "foo.a", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T04:48:42.522 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "mem_total_kb": "32494636", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "num_handles": "1", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "os": "Linux", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "pid": "2", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "realm_id": "", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "realm_name": "", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zone_id": "f87c8106-6d08-4ff5-be50-bc7389e33d34", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zone_name": "default", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zonegroup_id": "c27bb0de-7d96-4ff7-880a-a7fd7414e1ee", 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "zonegroup_name": "default" 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: }, 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: "task_status": {} 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.523 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.524 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.524 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.524 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: } 2024-03-26T04:48:42.524 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: 2024-03-26T04:48:42.524 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: attempt 2 of 20 2024-03-26T04:48:42.526 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (16328 ms) 2024-03-26T04:48:42.527 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (44828 ms total) 2024-03-26T04:48:42.527 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: 2024-03-26T04:48:42.527 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-26T04:48:42.527 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (44828 ms total) 2024-03-26T04:48:42.527 INFO:tasks.workunit.client.0.smithi012.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-26T04:48:42.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: 11.7 deep-scrub starts 2024-03-26T04:48:42.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: 11.7 deep-scrub ok 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: 11.0 deep-scrub starts 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: 11.0 deep-scrub ok 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1602716951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi012-78567-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4030158413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:42.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: 11.7 deep-scrub starts 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: 11.7 deep-scrub ok 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: 11.0 deep-scrub starts 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: 11.0 deep-scrub ok 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1602716951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi012-78567-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4030158413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:42.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:42.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: 11.7 deep-scrub starts 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: 11.7 deep-scrub ok 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: 11.0 deep-scrub starts 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: 11.0 deep-scrub ok 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1602716951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi012-78567-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]: dispatch 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4030158413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:42.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:43.542 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: Running main() from gmock_main.cc 2024-03-26T04:48:43.542 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [----------] Global test environment set-up. 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (10750 ms) 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (7297 ms) 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3998 ms) 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (4288 ms) 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (4592 ms) 2024-03-26T04:48:43.543 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (4556 ms) 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (10234 ms) 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (45715 ms total) 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [----------] Global test environment tear-down 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (45715 ms total) 2024-03-26T04:48:43.544 INFO:tasks.workunit.client.0.smithi012.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] Global test environment set-up. 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: 2024-03-26T04:47:57.495+0000 7f78d1f76780 -1 failed for service _ceph-mon._tcp 2024-03-26T04:48:43.568 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: 2024-03-26T04:47:57.495+0000 7f78d1f76780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (55 ms) 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2010 ms) 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2066 ms total) 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: started 0x55712680c600 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: started 0x557126b470d0 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: started 2 aios 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: waiting 0x55712680c600 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: waiting 0x557126b470d0 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: done. 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (12333 ms) 2024-03-26T04:48:43.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (12333 ms total) 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.Exec (103 ms) 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (7 ms) 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5930 ms) 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-26T04:48:43.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22635 ms) 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28681 ms total) 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [----------] Global test environment tear-down 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (46105 ms total) 2024-03-26T04:48:43.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: 11.3 deep-scrub starts 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: 11.3 deep-scrub ok 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: 11.4 deep-scrub starts 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: 11.4 deep-scrub ok 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: 11.1 deep-scrub starts 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: 11.1 deep-scrub ok 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: pgmap v121: 584 pgs: 96 creating+peering, 32 unknown, 456 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 17 MiB/s wr, 5 op/s 2024-03-26T04:48:43.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/248681934' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]': finished 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1637924383' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi012-78552-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]: dispatch 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:43.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]: dispatch 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: 11.3 deep-scrub starts 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: 11.3 deep-scrub ok 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: 11.4 deep-scrub starts 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: 11.4 deep-scrub ok 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: 11.1 deep-scrub starts 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: 11.1 deep-scrub ok 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: pgmap v121: 584 pgs: 96 creating+peering, 32 unknown, 456 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 17 MiB/s wr, 5 op/s 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/248681934' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:43.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]': finished 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1637924383' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi012-78552-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]: dispatch 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:43.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]: dispatch 2024-03-26T04:48:43.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: 11.3 deep-scrub starts 2024-03-26T04:48:43.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: 11.3 deep-scrub ok 2024-03-26T04:48:43.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: 11.4 deep-scrub starts 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: 11.4 deep-scrub ok 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: 11.1 deep-scrub starts 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: 11.1 deep-scrub ok 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: pgmap v121: 584 pgs: 96 creating+peering, 32 unknown, 456 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 17 MiB/s wr, 5 op/s 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/248681934' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-7"}]': finished 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1637924383' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi012-78552-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:43.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]: dispatch 2024-03-26T04:48:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:43.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]: dispatch 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1637924383' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi012-78552-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]': finished 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4235876160' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: pgmap v124: 584 pgs: 224 unknown, 360 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 286 B/s rd, 20 MiB/s wr, 2 op/s 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1637924383' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi012-78552-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]': finished 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4235876160' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:44.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:44.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:44.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: pgmap v124: 584 pgs: 224 unknown, 360 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 286 B/s rd, 20 MiB/s wr, 2 op/s 2024-03-26T04:48:44.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:44.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:44 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1637924383' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi012-78552-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4"}]': finished 2024-03-26T04:48:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:48:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]: dispatch 2024-03-26T04:48:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4235876160' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: pgmap v124: 584 pgs: 224 unknown, 360 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 286 B/s rd, 20 MiB/s wr, 2 op/s 2024-03-26T04:48:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:44.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:44 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:45.805 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]': finished 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.806 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]': finished 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.807 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-26T04:48:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi012-79023-4", "tierpool": "test-rados-api-smithi012-79023-6"}]': finished 2024-03-26T04:48:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi012-78567-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/473876375' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:48:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:48:45.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout:api_watch_notify_pp: flushed 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3032 ms) 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: trying... 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442547920 notify_id 433791696898 notifier_gid 25387 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: timed out 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: flushing 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: flushed 2024-03-26T04:48:46.649 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3122 ms) 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: List watches 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify2 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442547920 notify_id 442381631492 notifier_gid 25387 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify2 done 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: watch_check 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: unwatch2 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: flushing 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: done 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3036 ms) 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: List watches 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify2 2024-03-26T04:48:46.650 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: handle_notify cookie 93844442547920 notify_id 455266533386 notifier_gid 25387 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: notify2 done 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: watch_check 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: unwatch2 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: flushing 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: done 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3040 ms) 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18463 ms total) 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (49121 ms total) 2024-03-26T04:48:46.651 INFO:tasks.workunit.client.0.smithi012.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:48:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: 11.8 deep-scrub starts 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: 11.8 deep-scrub ok 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3179171012' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: 11.8 deep-scrub starts 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: 11.8 deep-scrub ok 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T04:48:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3179171012' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:46.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: 11.8 deep-scrub starts 2024-03-26T04:48:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: 11.8 deep-scrub ok 2024-03-26T04:48:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi012-79023-6", "pool2": "test-rados-api-smithi012-79023-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:48:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T04:48:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3179171012' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:46.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: pgmap v127: 484 pgs: 128 unknown, 356 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: 11.6 deep-scrub starts 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: 11.9 deep-scrub starts 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: 11.9 deep-scrub ok 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: 11.6 deep-scrub ok 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1083017026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]': finished 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]: dispatch 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: pgmap v127: 484 pgs: 128 unknown, 356 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:48.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: 11.6 deep-scrub starts 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: 11.9 deep-scrub starts 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: 11.9 deep-scrub ok 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: 11.6 deep-scrub ok 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1083017026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]': finished 2024-03-26T04:48:48.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]: dispatch 2024-03-26T04:48:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: pgmap v127: 484 pgs: 128 unknown, 356 active+clean; 217 MiB data, 994 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: 11.6 deep-scrub starts 2024-03-26T04:48:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: 11.9 deep-scrub starts 2024-03-26T04:48:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: 11.9 deep-scrub ok 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: 11.6 deep-scrub ok 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi012-78552-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1083017026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-9"}]': finished 2024-03-26T04:48:48.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78901-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi012-78567-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:48.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]: dispatch 2024-03-26T04:48:48.538 INFO:tasks.workunit.client.0.smithi012.stdout: misc: Running main() from gmock_main.cc 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [----------] Global test environment set-up. 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.Version (9549 ms) 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2078 ms) 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.LongName (5673 ms) 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (4229 ms) 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3694 ms) 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-26T04:48:48.539 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4728 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.Exec (4612 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3308 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3433 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.BigObject (4455 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2011 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2988 ms) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [----------] 12 tests from NeoRadosMisc (50758 ms total) 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: 2024-03-26T04:48:48.540 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [----------] Global test environment tear-down 2024-03-26T04:48:48.541 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [==========] 12 tests from 1 test suite ran. (50759 ms total) 2024-03-26T04:48:48.541 INFO:tasks.workunit.client.0.smithi012.stdout: misc: [ PASSED ] 12 tests. 2024-03-26T04:48:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T04:48:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]: dispatch 2024-03-26T04:48:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: pgmap v130: 516 pgs: 160 unknown, 356 active+clean; 217 MiB data, 996 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]': finished 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/956493656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi012-78552-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]: dispatch 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: pgmap v130: 516 pgs: 160 unknown, 356 active+clean; 217 MiB data, 996 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]': finished 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/956493656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi012-78552-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:49.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:48:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T04:48:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]: dispatch 2024-03-26T04:48:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: pgmap v130: 516 pgs: 160 unknown, 356 active+clean; 217 MiB data, 996 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:48:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-9", "mode": "writeback"}]': finished 2024-03-26T04:48:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:48:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/956493656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi012-78552-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:49.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:48:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/956493656' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi012-78552-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache", "force_nonempty":""}]': finished 2024-03-26T04:48:50.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3175426414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78901-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:50.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/956493656' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi012-78552-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache", "force_nonempty":""}]': finished 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3175426414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78901-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:50.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/956493656' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi012-78552-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache", "force_nonempty":""}]': finished 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3175426414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78901-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]: dispatch 2024-03-26T04:48:50.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: pgmap v133: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 4 op/s 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78901-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]': finished 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:51.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache"}]: dispatch 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: pgmap v133: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 4 op/s 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78901-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]': finished 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T04:48:51.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache"}]: dispatch 2024-03-26T04:48:51.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: pgmap v133: 580 pgs: 224 unknown, 356 active+clean; 217 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 4 op/s 2024-03-26T04:48:51.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:51.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:51.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:48:51.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78901-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:51.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-9"}]': finished 2024-03-26T04:48:51.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:51.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T04:48:51.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache"}]: dispatch 2024-03-26T04:48:52.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache"}]': finished 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2860416513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi012-78552-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: pgmap v136: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:52.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache"}]': finished 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2860416513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi012-78552-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: pgmap v136: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:52.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:52.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:52.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/487286240' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78901-10", "tierpool":"test-rados-api-smithi012-78901-10-cache"}]': finished 2024-03-26T04:48:52.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T04:48:52.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2860416513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi012-78552-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:52.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: pgmap v136: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-26T04:48:52.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:52.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:52.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:52.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:48:53.810 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: Running main() from gmock_main.cc 2024-03-26T04:48:53.810 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-26T04:48:53.810 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [----------] Global test environment set-up. 2024-03-26T04:48:53.810 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-26T04:48:53.810 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-26T04:48:53.810 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolList (9840 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (7456 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2969 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3118 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (6877 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6906 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5657 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5235 ms) 2024-03-26T04:48:53.811 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-26T04:48:53.812 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8261 ms) 2024-03-26T04:48:53.812 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [----------] 9 tests from LibRadosPools (56319 ms total) 2024-03-26T04:48:53.812 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: 2024-03-26T04:48:53.812 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [----------] Global test environment tear-down 2024-03-26T04:48:53.812 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (56319 ms total) 2024-03-26T04:48:53.812 INFO:tasks.workunit.client.0.smithi012.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2860416513' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi012-78552-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/254609793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi012-78928-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2860416513' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi012-78552-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T04:48:54.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/254609793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi012-78928-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:54.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:54.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2860416513' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi012-78552-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:54.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:48:54.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T04:48:54.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/254609793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi012-78928-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:48:54.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:54.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: pgmap v139: 420 pgs: 128 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi012-78928-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1264944986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: pgmap v139: 420 pgs: 128 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi012-78928-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1264944986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:55.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi012-78567-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: pgmap v139: 420 pgs: 128 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:48:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:48:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:55.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi012-78928-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:48:55.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:48:55.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1264944986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:55.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T04:48:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:48:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1905858083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:48:56.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:56.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:56.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:56.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:56.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:56.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T04:48:56.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:48:56.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1905858083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:48:56.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:48:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi012-78552-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:48:56.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T04:48:56.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:48:56.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1905858083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:48:56.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:56.558 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: Running main() from gmock_main.cc 2024-03-26T04:48:56.558 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-26T04:48:56.558 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [----------] Global test environment set-up. 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (14036 ms) 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (5365 ms) 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2655 ms) 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3129 ms) 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3503 ms) 2024-03-26T04:48:56.559 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3995 ms) 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (4165 ms) 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3385 ms) 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3334 ms) 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3175 ms) 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-26T04:48:56.560 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2987 ms) 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3021 ms) 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3262 ms) 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2740 ms) 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (58752 ms total) 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [----------] Global test environment tear-down 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (58752 ms total) 2024-03-26T04:48:56.561 INFO:tasks.workunit.client.0.smithi012.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: pgmap v142: 492 pgs: 200 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]': finished 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]: dispatch 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T04:48:57.124 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]: dispatch 2024-03-26T04:48:57.154 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:48:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: pgmap v142: 492 pgs: 200 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]': finished 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]: dispatch 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]: dispatch 2024-03-26T04:48:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: pgmap v142: 492 pgs: 200 unknown, 292 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-11"}]': finished 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi012-78567-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]: dispatch 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T04:48:57.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]: dispatch 2024-03-26T04:48:58.333 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:58.333 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:58 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:58.333 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:58 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]': finished 2024-03-26T04:48:58.333 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:58 smithi186 ceph-mon[33438]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T04:48:58.333 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1585477589' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi012-78552-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]': finished 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[29863]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1585477589' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi012-78552-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-11", "mode": "writeback"}]': finished 2024-03-26T04:48:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[41141]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T04:48:58.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1585477589' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi012-78552-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: pgmap v145: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1585477589' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi012-78552-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3121617717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi012-78567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: pgmap v145: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1585477589' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi012-78552-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3121617717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi012-78567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:48:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:59.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: pgmap v145: 396 pgs: 32 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-26T04:48:59.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:48:59.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1585477589' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi012-78552-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:48:59.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:48:59.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3121617717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi012-78567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:48:59.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:48:59.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:48:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3121617717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi012-78567-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:49:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[29863]: pgmap v148: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3121617717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi012-78567-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:49:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:00 smithi012 ceph-mon[41141]: pgmap v148: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-26T04:49:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3121617717' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi012-78567-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:49:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:49:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]: dispatch 2024-03-26T04:49:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:00 smithi186 ceph-mon[33438]: pgmap v148: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.5 KiB/s wr, 7 op/s 2024-03-26T04:49:01.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]': finished 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[29863]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/114769965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi012-78552-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]': finished 2024-03-26T04:49:01.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[41141]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T04:49:01.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/114769965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi012-78552-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:01.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:01.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:01 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:01.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-11"}]': finished 2024-03-26T04:49:01.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:01 smithi186 ceph-mon[33438]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T04:49:01.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/114769965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi012-78552-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:02.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/114769965' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi012-78552-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:02.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[29863]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T04:49:02.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3414493671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[29863]: pgmap v151: 428 pgs: 96 unknown, 32 creating+peering, 300 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:49:02.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:02.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/114769965' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi012-78552-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:02.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[41141]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T04:49:02.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3414493671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:02 smithi012 ceph-mon[41141]: pgmap v151: 428 pgs: 96 unknown, 32 creating+peering, 300 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:49:02.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:49:02.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:02.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/114769965' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi012-78552-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:02.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-mon[33438]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T04:49:02.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3414493671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:02.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:02 smithi186 ceph-mon[33438]: pgmap v151: 428 pgs: 96 unknown, 32 creating+peering, 300 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:49:03.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:03.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:03.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T04:49:03.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:03.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi012-78567-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:03.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:03.565 INFO:tasks.workunit.client.0.smithi012.stdout: io: Running main() from gmock_main.cc 2024-03-26T04:49:03.565 INFO:tasks.workunit.client.0.smithi012.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-26T04:49:03.565 INFO:tasks.workunit.client.0.smithi012.stdout: io: [----------] Global test environment set-up. 2024-03-26T04:49:03.565 INFO:tasks.workunit.client.0.smithi012.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-26T04:49:03.565 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-26T04:49:03.565 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.Limits (10831 ms) 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (10176 ms) 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.ReadOp (5399 ms) 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.SparseRead (4587 ms) 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.RoundTrip (6907 ms) 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3427 ms) 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-26T04:49:03.566 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3460 ms) 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3013 ms) 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2980 ms) 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.Trunc (3004 ms) 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.Remove (3009 ms) 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2983 ms) 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-26T04:49:03.567 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.RmXattr (3017 ms) 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3022 ms) 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: [----------] 14 tests from NeoRadosIo (65815 ms total) 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: [----------] Global test environment tear-down 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: [==========] 14 tests from 1 test suite ran. (65815 ms total) 2024-03-26T04:49:03.568 INFO:tasks.workunit.client.0.smithi012.stdout: io: [ PASSED ] 14 tests. 2024-03-26T04:49:04.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2160507520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[29863]: pgmap v154: 396 pgs: 96 unknown, 300 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:04.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:49:04.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2160507520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:04.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:04.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:04 smithi012 ceph-mon[41141]: pgmap v154: 396 pgs: 96 unknown, 300 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:49:04.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:04.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:04.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:04.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:49:04.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2160507520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:04.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:04.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:04 smithi186 ceph-mon[33438]: pgmap v154: 396 pgs: 96 unknown, 300 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:49:05.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[29863]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3135388373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:05.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:05.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[41141]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:49:05.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3135388373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:05.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:05.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:49:05.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:05.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi012-78552-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:05.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:05 smithi186 ceph-mon[33438]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:49:05.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3135388373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:05.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:05.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:49:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:49:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[29863]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T04:49:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:49:06.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[29863]: pgmap v157: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:06.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:06.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:06.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:49:06.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[41141]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T04:49:06.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:49:06.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-mon[41141]: pgmap v157: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:06.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:06.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi012-78567-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:06.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:49:06.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:06 smithi186 ceph-mon[33438]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T04:49:06.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]: dispatch 2024-03-26T04:49:06.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:06 smithi186 ceph-mon[33438]: pgmap v157: 388 pgs: 96 unknown, 292 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:07.132 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:49:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:49:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3370457940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi012-78552-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:07.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3370457940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi012-78552-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:07.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2800627407' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi012-78928-10"}]': finished 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3370457940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi012-78552-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:07.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3370457940' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi012-78552-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4214410059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi012-78567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:08.402 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:08 smithi186 ceph-mon[33438]: pgmap v160: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 74 KiB/s wr, 74 op/s 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3370457940' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi012-78552-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4214410059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi012-78567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:08.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[29863]: pgmap v160: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 74 KiB/s wr, 74 op/s 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3370457940' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi012-78552-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4214410059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi012-78567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:08.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:08 smithi012 ceph-mon[41141]: pgmap v160: 420 pgs: 96 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 74 KiB/s wr, 74 op/s 2024-03-26T04:49:09.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4214410059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi012-78567-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[29863]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4214410059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi012-78567-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[41141]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:49:09.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:09.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:09.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4214410059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi012-78567-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:09.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:49:09.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:09.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:09 smithi186 ceph-mon[33438]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:49:09.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]': finished 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]: dispatch 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]: dispatch 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3419018557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:10.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[29863]: pgmap v163: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 KiB/s wr, 76 op/s 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]': finished 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]: dispatch 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]: dispatch 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3419018557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:10.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:10 smithi012 ceph-mon[41141]: pgmap v163: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 KiB/s wr, 76 op/s 2024-03-26T04:49:10.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:10.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi012-78928-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:10.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-13"}]': finished 2024-03-26T04:49:10.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:49:10.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]: dispatch 2024-03-26T04:49:10.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]: dispatch 2024-03-26T04:49:10.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3419018557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:10.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:10.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:10 smithi186 ceph-mon[33438]: pgmap v163: 396 pgs: 72 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 KiB/s wr, 76 op/s 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]': finished 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2"}]: dispatch 2024-03-26T04:49:11.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2068044349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]': finished 2024-03-26T04:49:11.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2"}]: dispatch 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2068044349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-13", "mode": "writeback"}]': finished 2024-03-26T04:49:11.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi012-78552-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2"}]: dispatch 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2068044349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:11.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.0"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.0"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.1"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.1"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.2"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.2"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.3"}]: dispatch 2024-03-26T04:49:12.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.3"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.4"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.4"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.5"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.5"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.6"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.6"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.7"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.7"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.8"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.8"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.9"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.9"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:12.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: 177.9 scrub starts 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: 177.9 scrub ok 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.0"}]: dispatch 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.0"}]: dispatch 2024-03-26T04:49:12.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.1"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.1"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.2"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.2"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.3"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.3"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.4"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.4"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.5"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.5"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.6"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.6"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.7"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.7"}]: dispatch 2024-03-26T04:49:12.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.8"}]: dispatch 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.8"}]: dispatch 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.9"}]: dispatch 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.9"}]: dispatch 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: 177.9 scrub starts 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: 177.9 scrub ok 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T04:49:12.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: 177.2 deep-scrub starts 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: 177.2 deep-scrub ok 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[29863]: pgmap v166: 364 pgs: 32 creating+peering, 32 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: 177.2 deep-scrub starts 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: 177.2 deep-scrub ok 2024-03-26T04:49:12.409 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:12 smithi012 ceph-mon[41141]: pgmap v166: 364 pgs: 32 creating+peering, 32 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T04:49:12.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:49:12.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.0"}]: dispatch 2024-03-26T04:49:12.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.0"}]: dispatch 2024-03-26T04:49:12.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.1"}]: dispatch 2024-03-26T04:49:12.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.1"}]: dispatch 2024-03-26T04:49:12.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.2"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.2"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.3"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.3"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.4"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.4"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.5"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.5"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.6"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.6"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.7"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.7"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.8"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.8"}]: dispatch 2024-03-26T04:49:12.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "177.9"}]: dispatch 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "177.9"}]: dispatch 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: 177.9 scrub starts 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: 177.9 scrub ok 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi012-78567-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi012-78756-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: 177.2 deep-scrub starts 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: 177.2 deep-scrub ok 2024-03-26T04:49:12.464 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:12 smithi186 ceph-mon[33438]: pgmap v166: 364 pgs: 32 creating+peering, 32 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T04:49:13.142 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: 177.7 scrub starts 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: 177.7 scrub ok 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: 177.6 scrub starts 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: 177.6 scrub ok 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: 177.4 scrub starts 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: 177.4 scrub ok 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1142642622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.143 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2344557640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.144 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.144 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.144 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:49:13.144 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: 177.7 scrub starts 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: 177.7 scrub ok 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: 177.6 scrub starts 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: 177.6 scrub ok 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: 177.4 scrub starts 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: 177.4 scrub ok 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:13.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1142642622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2344557640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: 177.7 scrub starts 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: 177.7 scrub ok 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:13.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: 177.6 scrub starts 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: 177.6 scrub ok 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: 177.4 scrub starts 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: 177.4 scrub ok 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1142642622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2344557640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:49:13.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:49:14.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: 177.5 scrub starts 2024-03-26T04:49:14.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: 177.5 scrub ok 2024-03-26T04:49:14.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: 177.8 scrub starts 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: 177.8 scrub ok 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: 177.3 scrub starts 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: 177.3 scrub ok 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3304750262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:49:14.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[41141]: pgmap v169: 428 pgs: 128 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: 177.5 scrub starts 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: 177.5 scrub ok 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: 177.8 scrub starts 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: 177.8 scrub ok 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: 177.3 scrub starts 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: 177.3 scrub ok 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3304750262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:14.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:49:14.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:14.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:49:14.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:14 smithi012 ceph-mon[29863]: pgmap v169: 428 pgs: 128 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T04:49:14.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: 177.5 scrub starts 2024-03-26T04:49:14.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: 177.5 scrub ok 2024-03-26T04:49:14.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: 177.8 scrub starts 2024-03-26T04:49:14.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: 177.8 scrub ok 2024-03-26T04:49:14.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: 177.3 scrub starts 2024-03-26T04:49:14.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: 177.3 scrub ok 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi012-78552-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78916-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3304750262' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4208583664' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:49:14.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:14 smithi186 ceph-mon[33438]: pgmap v169: 428 pgs: 128 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: 177.1 scrub starts 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: 177.1 scrub ok 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: 177.0 scrub starts 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: 177.0 scrub ok 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:49:15.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[29863]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: 177.1 scrub starts 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: 177.1 scrub ok 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: 177.0 scrub starts 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: 177.0 scrub ok 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:49:15.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:15 smithi012 ceph-mon[41141]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: 177.1 scrub starts 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: 177.1 scrub ok 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: 177.0 scrub starts 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: 177.0 scrub ok 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:15.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi012-78756-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:49:15.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:15 smithi186 ceph-mon[33438]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:49:15.569 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: seed 78756 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (2 ms) 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (1382 ms) 2024-03-26T04:49:15.570 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (44 ms) 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (16 ms) 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (7 ms) 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (4 ms) 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2024-03-26T04:49:15.571 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (14 ms) 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (6 ms) 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (9 ms) 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-26T04:49:15.572 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1584 ms) 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (345 ms) 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: done waiting, doing copies 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: done waiting 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60791 ms) 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4376 ms) 2024-03-26T04:49:15.573 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-26T04:49:15.574 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-26T04:49:15.574 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-26T04:49:15.574 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-26T04:49:15.574 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-26T04:49:15.574 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3684629691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:16.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[41141]: pgmap v172: 364 pgs: 96 unknown, 268 active+clean; 599 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3684629691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:16.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-mon[29863]: pgmap v172: 364 pgs: 96 unknown, 268 active+clean; 599 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:16.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:16.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T04:49:16.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3684629691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:16.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:16.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:16.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:16.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:16.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:16 smithi186 ceph-mon[33438]: pgmap v172: 364 pgs: 96 unknown, 268 active+clean; 599 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:17.146 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:49:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:49:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833894130' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833894130' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:17.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:17.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi012-78552-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78756-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833894130' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:17.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:18.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: pgmap v175: 332 pgs: 32 unknown, 300 active+clean; 600 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:18.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:18.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: pgmap v175: 332 pgs: 32 unknown, 300 active+clean; 600 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:18.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T04:49:18.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi012-78567-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: pgmap v175: 332 pgs: 32 unknown, 300 active+clean; 600 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T04:49:18.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:19.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079862497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78552-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:19.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:19.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:19.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:19.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:49:19.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:19.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079862497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78552-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:19.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78756-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079862497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78552-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:19.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.592 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP api_snapshots: Running main() from gmock_main.cc 2024-03-26T04:49:20.592 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-26T04:49:20.592 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (5625 ms) 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (6131 ms) 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (4237 ms) 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (4724 ms) 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (20717 ms total) 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (5836 ms) 2024-03-26T04:49:20.593 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (5443 ms) 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5992 ms) 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (17271 ms total) 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2980 ms) 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2013 ms) 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2011 ms) 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-26T04:49:20.594 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2009 ms) 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9014 ms total) 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4068 ms) 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4253 ms) 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8321 ms total) 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (83080 ms total) 2024-03-26T04:49:20.595 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-26T04:49:20.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:20.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:20.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079862497' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78552-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/873667482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[29863]: pgmap v178: 404 pgs: 112 unknown, 292 active+clean; 592 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 op/s 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:20.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079862497' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78552-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/873667482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:20 smithi012 ceph-mon[41141]: pgmap v178: 404 pgs: 112 unknown, 292 active+clean; 592 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 op/s 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi012-78916-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079862497' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78552-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1732914404' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/873667482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]: dispatch 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:20.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:20 smithi186 ceph-mon[33438]: pgmap v178: 404 pgs: 112 unknown, 292 active+clean; 592 KiB data, 1010 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 op/s 2024-03-26T04:49:21.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[29863]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:21.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:21.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[41141]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:49:21.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:21.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:21.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi012-78928-15"}]': finished 2024-03-26T04:49:21.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi012-78567-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:21.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:21.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:21 smithi186 ceph-mon[33438]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:49:21.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:49:22.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[29863]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1106282195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[29863]: pgmap v181: 372 pgs: 32 unknown, 340 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[41141]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1106282195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:22 smithi012 ceph-mon[41141]: pgmap v181: 372 pgs: 32 unknown, 340 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-26T04:49:22.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:22.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78756-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:22.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-mon[33438]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:49:22.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1106282195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:22.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:22 smithi186 ceph-mon[33438]: pgmap v181: 372 pgs: 32 unknown, 340 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-03-26T04:49:23.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:23.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:23.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:23.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[29863]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:49:23.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1697456251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi012-78567-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:23.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:23.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:23.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:23.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[41141]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:49:23.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1697456251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi012-78567-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:23.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:23.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:23.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi012-78552-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:23.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:23 smithi186 ceph-mon[33438]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:49:23.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1697456251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi012-78567-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: seed 78916 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2401 ms) 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-26T04:49:24.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (4588 ms) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3206 ms) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2102 ms) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4839 ms) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (17136 ms total) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (5735 ms) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (5590 ms) 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-26T04:49:24.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (7556 ms) 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3893 ms) 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2070 ms) 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1775-g5e70124b/rpm/el9/BUILD/ceph-19.0.0-1775-g5e70124b/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi012-78916-7 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17972 ms) 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (42816 ms total) 2024-03-26T04:49:24.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5067 ms) 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5075 ms total) 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2942 ms) 2024-03-26T04:49:24.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-26T04:49:24.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:24.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1697456251' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi012-78567-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:24.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[41141]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[41141]: pgmap v184: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1697456251' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi012-78567-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[29863]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:24.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:24 smithi012 ceph-mon[29863]: pgmap v184: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:49:24.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:24.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1697456251' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi012-78567-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:24.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:24 smithi186 ceph-mon[33438]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:49:24.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:24.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:24 smithi186 ceph-mon[33438]: pgmap v184: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:49:25.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/368632430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:25.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:49:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/368632430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:25.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]: dispatch 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/368632430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:25.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:26.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:26.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:26.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3650767895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[41141]: pgmap v187: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3650767895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-mon[29863]: pgmap v187: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2960833193' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78756-24"}]': finished 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi012-78552-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3650767895' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:26.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:26 smithi186 ceph-mon[33438]: pgmap v187: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:27.153 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:49:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:49:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:27.613 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:27.613 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:27.613 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:27 smithi186 ceph-mon[33438]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:49:27.613 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1091123340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi012-78756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:27.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:27.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:27.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[29863]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:49:27.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1091123340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi012-78756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:27.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:27.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi012-78567-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:27.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[41141]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:49:27.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1091123340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi012-78756-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:28.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1091123340' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi012-78756-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:28.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[29863]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T04:49:28.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2463683667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[29863]: pgmap v190: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1091123340' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi012-78756-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[41141]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2463683667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:28 smithi012 ceph-mon[41141]: pgmap v190: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:49:28.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:28.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1091123340' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi012-78756-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:28.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:28 smithi186 ceph-mon[33438]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T04:49:28.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2463683667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:28.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:28 smithi186 ceph-mon[33438]: pgmap v190: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:49:29.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:29.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:29.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[29863]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T04:49:29.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1902838956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:29.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:29.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:29.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:29.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[41141]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T04:49:29.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1902838956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:29.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:29.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:29.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi012-78552-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:29.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:29 smithi186 ceph-mon[33438]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T04:49:29.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1902838956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:29.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:30.590 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [----------] Global test environment set-up. 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (12676 ms) 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (7273 ms) 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (5246 ms) 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (10299 ms) 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-26T04:49:30.591 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (8078 ms) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (43572 ms total) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5265 ms) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5895 ms) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8007 ms) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6042 ms) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3960 ms) 2024-03-26T04:49:30.592 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi012-79821-11. 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: Waiting for snaps to purge. 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20046 ms) 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (49215 ms total) 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [----------] Global test environment tear-down 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (92787 ms total) 2024-03-26T04:49:30.593 INFO:tasks.workunit.client.0.smithi012.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-26T04:49:30.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:30.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:30.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T04:49:30.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:30.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1321115515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi012-78756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[29863]: pgmap v193: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1321115515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi012-78756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:30.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:30 smithi012 ceph-mon[41141]: pgmap v193: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi012-78567-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1321115515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi012-78756-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:30.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:30 smithi186 ceph-mon[33438]: pgmap v193: 356 pgs: 64 unknown, 292 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:31.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:31.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:31.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1321115515' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi012-78756-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2711004019' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:31.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1321115515' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi012-78756-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:31.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:31.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T04:49:31.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2711004019' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:31.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:31.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1321115515' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi012-78756-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1568454525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2711004019' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]: dispatch 2024-03-26T04:49:31.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.361 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:49:32.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:32.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:32.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:32.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T04:49:32.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2588572055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: pgmap v196: 324 pgs: 32 creating+peering, 32 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2588572055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:32.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: pgmap v196: 324 pgs: 32 creating+peering, 32 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:49:32.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:32.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi012-78916-16"}]': finished 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi012-78552-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2588572055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: pgmap v196: 324 pgs: 32 creating+peering, 32 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:49:32.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:32.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:33.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:33.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:33.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:33.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/310069660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T04:49:33.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:33.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/310069660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:33.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:33.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi012-78567-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/310069660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:33.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:34.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:34.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:34.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[29863]: pgmap v199: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[29863]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1491436887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[41141]: pgmap v199: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[41141]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1491436887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:34.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:34.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:34.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi012-78756-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:34.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:34 smithi186 ceph-mon[33438]: pgmap v199: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:34.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:34 smithi186 ceph-mon[33438]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T04:49:34.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1491436887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:34.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:35.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:35.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:35.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/82884267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:35.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/82884267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:35.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:35.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi012-78552-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/82884267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:35.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:36.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:36.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:36.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:36.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[29863]: pgmap v202: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:36.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-mon[41141]: pgmap v202: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi012-78567-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:36.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:36 smithi186 ceph-mon[33438]: pgmap v202: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:37.153 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:49:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:49:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:37.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:37.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[29863]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T04:49:37.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2012841459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:37.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:37.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:37.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[41141]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T04:49:37.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2012841459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:37.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:37.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:37 smithi186 ceph-mon[33438]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T04:49:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2012841459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:37.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:38.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/119119998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[29863]: pgmap v205: 340 pgs: 72 unknown, 268 active+clean; 600 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:38.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:38.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T04:49:38.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/119119998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:38 smithi012 ceph-mon[41141]: pgmap v205: 340 pgs: 72 unknown, 268 active+clean; 600 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T04:49:38.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi012-78756-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi012-78552-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/119119998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:38.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:38 smithi186 ceph-mon[33438]: pgmap v205: 340 pgs: 72 unknown, 268 active+clean; 600 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T04:49:39.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:39.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:39.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[29863]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T04:49:39.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:39.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:39.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi012-78552-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:39.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:39.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:39.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[41141]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T04:49:39.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:39.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:39.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi012-78552-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:39.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:39.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi012-78567-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:39.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:39 smithi186 ceph-mon[33438]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T04:49:39.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:39.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:39.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi012-78552-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:40.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:40.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi012-78552-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:40.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T04:49:40.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi012-78552-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:40.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[29863]: pgmap v208: 268 pgs: 268 active+clean; 600 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi012-78552-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi012-78552-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:40.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:40 smithi012 ceph-mon[41141]: pgmap v208: 268 pgs: 268 active+clean; 600 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T04:49:40.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi012-78552-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T04:49:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi012-78552-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:40.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:40 smithi186 ceph-mon[33438]: pgmap v208: 268 pgs: 268 active+clean; 600 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T04:49:41.631 INFO:tasks.workunit.client.0.smithi012.stdout: (68603 ms total) 2024-03-26T04:49:41.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:41.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (309 ms) 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (309 ms total) 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (42 ms) 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (46 ms total) 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-26T04:49:41.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (39 ms) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (42 ms total) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (30 ms) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (2 ms) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (32 ms total) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1039 ms) 2024-03-26T04:49:41.633 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1039 ms total) 2024-03-26T04:49:41.634 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: 2024-03-26T04:49:41.634 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-26T04:49:41.634 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104164 ms total) 2024-03-26T04:49:41.634 INFO:tasks.workunit.client.0.smithi012.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-26T04:49:41.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:41.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:41.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:41.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T04:49:41.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3184395122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T04:49:41.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3184395122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:41.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:41.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:41.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:41.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:41.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1314599739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:41.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T04:49:41.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3184395122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]: dispatch 2024-03-26T04:49:41.713 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:42.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:49:42.653 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi012-78552-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:49:42.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[29863]: pgmap v211: 308 pgs: 8 unknown, 32 creating+peering, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi012-78552-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:49:42.655 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:42 smithi012 ceph-mon[41141]: pgmap v211: 308 pgs: 8 unknown, 32 creating+peering, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-26T04:49:42.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi012-78552-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi012-78756-36"}]': finished 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi012-78567-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:49:42.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:42 smithi186 ceph-mon[33438]: pgmap v211: 308 pgs: 8 unknown, 32 creating+peering, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-26T04:49:43.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:43.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:43.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[41141]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[29863]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T04:49:43.654 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:43.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:43.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:43 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:43.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:49:43.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:43 smithi186 ceph-mon[33438]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T04:49:43.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:44.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T04:49:44.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:44.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:44.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3987430110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: pgmap v214: 300 pgs: 32 unknown, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3987430110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:44.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: pgmap v214: 300 pgs: 32 unknown, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-26T04:49:44.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:44 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:49:44.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T04:49:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3987430110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: pgmap v214: 300 pgs: 32 unknown, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 4 op/s 2024-03-26T04:49:44.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:44 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: Running main() from gmock_main.cc 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [----------] Global test environment set-up. 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.TooBig (9638 ms) 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-26T04:49:45.655 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (6990 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (6849 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4481 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (5191 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (5133 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (5709 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3191 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2980 ms) 2024-03-26T04:49:45.656 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2994 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3284 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2752 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2975 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3019 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.Flush (3008 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3223 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2752 ms) 2024-03-26T04:49:45.657 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3020 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3267 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2773 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2927 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3029 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3014 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3017 ms) 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-26T04:49:45.658 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3009 ms) 2024-03-26T04:49:45.659 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-26T04:49:45.659 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3013 ms) 2024-03-26T04:49:45.659 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [----------] 26 tests from LibRadosAio (101238 ms total) 2024-03-26T04:49:45.659 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: 2024-03-26T04:49:45.659 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-26T04:49:45.659 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-26T04:49:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T04:49:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: pgmap v216: 300 pgs: 32 unknown, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: pgmap v216: 300 pgs: 32 unknown, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:45.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:45 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi012-78567-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]: dispatch 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: pgmap v216: 300 pgs: 32 unknown, 268 active+clean; 616 KiB data, 582 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:45.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:45 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:46.903 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:49:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:46.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-26T04:49:46.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:46.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:46.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:46.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-26T04:49:46.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4241903620' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi012-78552-27"}]': finished 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:46.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:48.064 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]': finished 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2184370427' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: pgmap v219: 300 pgs: 32 unknown, 268 active+clean; 609 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.5 KiB/s wr, 27 op/s 2024-03-26T04:49:48.065 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]': finished 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2184370427' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: pgmap v219: 300 pgs: 32 unknown, 268 active+clean; 609 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.5 KiB/s wr, 27 op/s 2024-03-26T04:49:48.066 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-13"}]': finished 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi012-78552-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2184370427' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: pgmap v219: 300 pgs: 32 unknown, 268 active+clean; 609 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 2.5 KiB/s wr, 27 op/s 2024-03-26T04:49:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi012-78567-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:49.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T04:49:49.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:49.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:49.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:49.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:49.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:49.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:49.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi012-78552-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]: dispatch 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:49.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:49.537 INFO:tasks.workunit.client.0.smithi012.stdout:hotsECPP.SnapRemovePP (2012 ms) 2024-03-26T04:49:49.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-26T04:49:49.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2016 ms) 2024-03-26T04:49:49.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-26T04:49:49.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2010 ms) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8980 ms total) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4046 ms) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3999 ms) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4026 ms) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12071 ms total) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (112017 ms total) 2024-03-26T04:49:49.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-26T04:49:49.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-26T04:49:49.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: pgmap v222: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1926604017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:50.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: pgmap v222: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1926604017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:50.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:50.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: pgmap v222: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3266715272' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi012-78916-21"}]': finished 2024-03-26T04:49:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T04:49:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1926604017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:50.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:51.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:51.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:51.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T04:49:51.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:51.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:51.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:51.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:51.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:51.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi012-78567-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:51.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T04:49:51.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:51.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:51.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:51.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:49:51.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:51.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-26T04:49:51.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-26T04:49:51.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-26T04:49:51.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-26T04:49:51.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-26T04:49:51.535 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (9957 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (21455 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (10270 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3496 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3007 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2969 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3007 ms) 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-26T04:49:51.536 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3011 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2985 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3019 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3008 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3020 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2961 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3014 ms) 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-26T04:49:51.537 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3011 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3274 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2758 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2945 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3016 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3029 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3261 ms) 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-26T04:49:51.538 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3016 ms) 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2757 ms) 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3019 ms) 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3016 ms) 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2882 ms) 2024-03-26T04:49:51.539 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: pgmap v225: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:52.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: pgmap v225: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:52.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:52.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:52.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:52.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:52.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:49:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: pgmap v225: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 572 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2624105202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]: dispatch 2024-03-26T04:49:52.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]': finished 2024-03-26T04:49:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T04:49:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/871955764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi012-78567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi012-78552-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[41141]: pgmap v228: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]': finished 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:53.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/871955764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi012-78567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi012-78552-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:53.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:53 smithi012 ceph-mon[29863]: pgmap v228: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-15"}]': finished 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi012-78552-28"}]': finished 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/871955764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi012-78567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]: dispatch 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]: dispatch 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:53.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi012-78552-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:49:53.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:53.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:53 smithi186 ceph-mon[33438]: pgmap v228: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/871955764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi012-78567-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]': finished 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi012-78552-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi012-78552-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:55.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/871955764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi012-78567-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]': finished 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi012-78552-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi012-78552-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:55.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:55.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/871955764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi012-78567-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-15", "mode": "writeback"}]': finished 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi012-78552-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi012-78552-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:49:55.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:56.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T04:49:56.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:56.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: pgmap v231: 260 pgs: 260 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:49:56.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi012-78552-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2761177050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:56.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: pgmap v231: 260 pgs: 260 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi012-78552-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2761177050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:56.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:56.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T04:49:56.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: pgmap v231: 260 pgs: 260 active+clean; 585 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi012-78552-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2761177050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:56.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]: dispatch 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]': finished 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:57.113 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:49:57.153 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:49:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]': finished 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi012-78567-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-15"}]': finished 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:49:58.396 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: pgmap v234: 300 pgs: 40 creating+peering, 1 peering, 259 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:49:58.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:49:58.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T04:49:58.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:58.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:58.398 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:58.398 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: pgmap v234: 300 pgs: 40 creating+peering, 1 peering, 259 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: pgmap v234: 300 pgs: 40 creating+peering, 1 peering, 259 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:49:58.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T04:49:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:58.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:49:59.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:49:59.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1805244401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:49:59 smithi186 ceph-mon[33438]: pgmap v237: 292 pgs: 64 unknown, 1 peering, 227 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:59.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:49:59.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:49:59.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:49:59.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T04:49:59.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1805244401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[29863]: pgmap v237: 292 pgs: 64 unknown, 1 peering, 227 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi012-78640-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1805244401' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]: dispatch 2024-03-26T04:49:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:49:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:49:59.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:49:59 smithi012 ceph-mon[41141]: pgmap v237: 292 pgs: 64 unknown, 1 peering, 227 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:00.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:00.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:00.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: Health detail: HEALTH_WARN Reduced data availability: 1 pg peering; 3 pool(s) do not have an application enabled 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: pg 22.1f is stuck peering for 111s, current state peering, last acting [5,6,4] 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: application not enabled on pool 'test-rados-api-smithi012-78824-17' 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: application not enabled on pool 'RoundTripCmpExtPP2_smithi012-78567-32' 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[29863]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:00.405 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:00.406 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: Health detail: HEALTH_WARN Reduced data availability: 1 pg peering; 3 pool(s) do not have an application enabled 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: pg 22.1f is stuck peering for 111s, current state peering, last acting [5,6,4] 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: application not enabled on pool 'test-rados-api-smithi012-78824-17' 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: application not enabled on pool 'RoundTripCmpExtPP2_smithi012-78567-32' 2024-03-26T04:50:00.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:00 smithi012 ceph-mon[41141]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:50:00.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:00.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4085830475' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi012-78552-29"}]': finished 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi012-78567-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:00.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: Health detail: HEALTH_WARN Reduced data availability: 1 pg peering; 3 pool(s) do not have an application enabled 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: pg 22.1f is stuck peering for 111s, current state peering, last acting [5,6,4] 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: application not enabled on pool 'test-rados-api-smithi012-78824-17' 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: application not enabled on pool 'RoundTripCmpExtPP2_smithi012-78567-32' 2024-03-26T04:50:00.463 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:00 smithi186 ceph-mon[33438]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:50:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T04:50:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[29863]: pgmap v240: 260 pgs: 32 unknown, 1 peering, 227 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:01.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:01.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:01.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:01 smithi012 ceph-mon[41141]: pgmap v240: 260 pgs: 32 unknown, 1 peering, 227 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi012-78552-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:01.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:01 smithi186 ceph-mon[33438]: pgmap v240: 260 pgs: 32 unknown, 1 peering, 227 active+clean; 585 KiB data, 545 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:02.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]': finished 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]: dispatch 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1893893880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]: dispatch 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]': finished 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[29863]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]': finished 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]: dispatch 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1893893880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]: dispatch 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:02.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]': finished 2024-03-26T04:50:02.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:02.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:02 smithi012 ceph-mon[41141]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T04:50:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-17"}]': finished 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]: dispatch 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1893893880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]: dispatch 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi012-78552-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-17", "mode": "writeback"}]': finished 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi012-78567-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:02 smithi186 ceph-mon[33438]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: pgmap v243: 300 pgs: 40 unknown, 260 active+clean; 585 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1893893880' entity='client.admin' cmd=[{ 2024-03-26T04:50:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "prefix": "osd pool set", 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "var": "eio", 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "val": "true" 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: }]: dispatch 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{ 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "prefix": "osd pool set", 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "var": "eio", 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "val": "true" 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: }]: dispatch 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T04:50:03.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{ 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "prefix": "osd pool set", 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "var": "eio", 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: "val": "true" 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: }]': finished 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:03.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: pgmap v243: 300 pgs: 40 unknown, 260 active+clean; 585 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1893893880' entity='client.admin' cmd=[{ 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "prefix": "osd pool set", 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "var": "eio", 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "val": "true" 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: }]: dispatch 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{ 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "prefix": "osd pool set", 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "var": "eio", 2024-03-26T04:50:03.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "val": "true" 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: }]: dispatch 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{ 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "prefix": "osd pool set", 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "var": "eio", 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: "val": "true" 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: }]': finished 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T04:50:03.908 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: pgmap v243: 300 pgs: 40 unknown, 260 active+clean; 585 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:50:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1893893880' entity='client.admin' cmd=[{ 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "prefix": "osd pool set", 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "var": "eio", 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "val": "true" 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: }]: dispatch 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{ 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "prefix": "osd pool set", 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "var": "eio", 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "val": "true" 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: }]: dispatch 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:03.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{ 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "prefix": "osd pool set", 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "pool": "PoolEIOFlag_smithi012-78567-33", 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "var": "eio", 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: "val": "true" 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: }]': finished 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T04:50:03.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]: dispatch 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]': finished 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]': finished 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:05.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:05.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T04:50:05.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:05.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:05.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-17"}]': finished 2024-03-26T04:50:05.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T04:50:05.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:05.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: pgmap v246: 260 pgs: 260 active+clean; 585 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T04:50:06.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1443457312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi012-78567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: pgmap v246: 260 pgs: 260 active+clean; 585 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T04:50:06.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T04:50:06.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1443457312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi012-78567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:06.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:06.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: pgmap v246: 260 pgs: 260 active+clean; 585 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi012-78640-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1443457312' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi012-78567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1685178841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:06.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]: dispatch 2024-03-26T04:50:07.153 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:50:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1443457312' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi012-78567-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1443457312' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi012-78567-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:07.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1649800956' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:50:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1443457312' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi012-78567-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi012-78552-30"}]': finished 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:07.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: pgmap v249: 291 pgs: 32 unknown, 32 creating+peering, 2 peering, 225 active+clean; 585 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: pgmap v249: 291 pgs: 32 unknown, 32 creating+peering, 2 peering, 225 active+clean; 585 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: pgmap v249: 291 pgs: 32 unknown, 32 creating+peering, 2 peering, 225 active+clean; 585 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi012-78552-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T04:50:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3092497824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi012-78567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:09.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T04:50:09.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:09.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3092497824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi012-78567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi012-78640-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:09.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3092497824' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi012-78567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: pgmap v252: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]': finished 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3092497824' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi012-78567-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]: dispatch 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]: dispatch 2024-03-26T04:50:10.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: pgmap v252: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]': finished 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3092497824' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi012-78567-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]: dispatch 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T04:50:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]: dispatch 2024-03-26T04:50:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: pgmap v252: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T04:50:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi012-78552-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-19"}]': finished 2024-03-26T04:50:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3092497824' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi012-78567-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]: dispatch 2024-03-26T04:50:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T04:50:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]: dispatch 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]': finished 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[29863]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:11.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:11.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]': finished 2024-03-26T04:50:11.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[41141]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T04:50:11.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:11.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:10 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi012-78640-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-19", "mode": "writeback"}]': finished 2024-03-26T04:50:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:10 smithi186 ceph-mon[33438]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T04:50:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:11.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAiexpected=5 2024-03-26T04:50:11.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:5c6b0b28:::7:head 2024-03-26T04:50:11.627 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:5c6b0b28:::7:head expected=22:5c6b0b28:::7:head 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:5c6b0b28:::7:head -> 7 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=7 expected=7 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:89d3ae78:::11:head 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:89d3ae78:::11:head expected=22:89d3ae78:::11:head 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:89d3ae78:::11:head -> 11 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=11 expected=11 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:863748b0:::15:head 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:863748b0:::15:head expected=22:863748b0:::15:head 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:863748b0:::15:head -> 15 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=15 expected=15 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:6cac518f:::0:head 2024-03-26T04:50:11.628 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:6cac518f:::0:head expected=22:6cac518f:::0:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:6cac518f:::0:head -> 0 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=0 expected=0 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:62a1935d:::14:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:62a1935d:::14:head expected=22:62a1935d:::14:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:62a1935d:::14:head -> 14 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=14 expected=14 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:f905c69b:::2:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:f905c69b:::2:head expected=22:f905c69b:::2:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:f905c69b:::2:head -> 2 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=2 expected=2 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:e960b815:::9:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:e960b815:::9:head expected=22:e960b815:::9:head 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:e960b815:::9:head -> 9 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=9 expected=9 2024-03-26T04:50:11.629 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : seek to 22:566253c9:::13:head 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : cursor()=22:566253c9:::13:head expected=22:566253c9:::13:head 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: > 22:566253c9:::13:head -> 13 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: : entry=13 expected=13 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (222 ms) 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (108778 ms) 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: split 0/5 -> MIN 22:33333333::::head 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: split 1/5 -> 22:33333333::::head 22:66666666::::head 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: split 2/5 -> 22:66666666::::head 22:99999999::::head 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: split 3/5 -> 22:99999999::::head 22:cccccccc::::head 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: split 4/5 -> 22:cccccccc::::head MAX 2024-03-26T04:50:11.630 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (10011 ms) 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] 7 tests from LibRadosList (120160 ms total) 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1034 ms) 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo1 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo2 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo3 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T04:50:11.631 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo4 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo5 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo1 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset foo6,foo7 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo7 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: foo6 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns1:foo4 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns1:foo5 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns2:foo7 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns2:foo6 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: ns1:foo1 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: :foo1 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: :foo2 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: :foo3 2024-03-26T04:50:11.632 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (34 ms) 2024-03-26T04:50:12.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: pgmap v255: 244 pgs: 48 unknown, 196 active+clean; 583 KiB data, 558 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:12.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:12.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:12.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:12.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T04:50:12.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:12.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:12.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3348519390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:12.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:12.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:12.156 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: pgmap v255: 244 pgs: 48 unknown, 196 active+clean; 583 KiB data, 558 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3348519390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:12.157 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:12.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: pgmap v255: 244 pgs: 48 unknown, 196 active+clean; 583 KiB data, 558 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]: dispatch 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3348519390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:12.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]': finished 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:13.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:13.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:13.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:13.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:13.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]': finished 2024-03-26T04:50:13.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:13.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:13.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T04:50:13.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:13.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:13.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:13.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:13.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:13.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-19"}]': finished 2024-03-26T04:50:13.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:13.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:13.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T04:50:13.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2546479209' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:13.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:13.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]: dispatch 2024-03-26T04:50:13.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: pgmap v258: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:50:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T04:50:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2496227883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:14.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:14.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi012-78552-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:14.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: pgmap v258: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:50:14.155 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2496227883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:14.156 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:14.157 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi012-78552-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: pgmap v258: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi012-78552-31"}]': finished 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1054730612' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2496227883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]: dispatch 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:14.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi012-78552-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi012-78552-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi012-78552-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[29863]: pgmap v261: 292 pgs: 96 unknown, 196 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi012-78552-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi012-78552-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:15 smithi012 ceph-mon[41141]: pgmap v261: 292 pgs: 96 unknown, 196 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi012-78640-2"}]': finished 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi012-78552-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi012-78552-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:15.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:15 smithi186 ceph-mon[33438]: pgmap v261: 292 pgs: 96 unknown, 196 active+clean; 583 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:16.903 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:50:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T04:50:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2738760567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/345595402' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2738760567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/345595402' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T04:50:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2738760567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/345595402' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:16.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:16.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 1 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 10 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 13 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 7 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 14 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 0 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 15 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 11 0 2024-03-26T04:50:17.613 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 5 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 8 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 6 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 3 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 4 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 12 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 9 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 2 0 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (46 ms) 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] 3 tests from LibRadosListEC (1114 ms total) 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-26T04:50:17.614 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3030 ms) 2024-03-26T04:50:17.615 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] 1 test from LibRadosListNP (3030 ms total) 2024-03-26T04:50:17.615 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: 2024-03-26T04:50:17.615 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [----------] Global test environment tear-down 2024-03-26T04:50:17.615 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [==========] 11 tests from 3 test suites ran. (140186 ms total) 2024-03-26T04:50:17.615 INFO:tasks.workunit.client.0.smithi012.stdout: api_list: [ PASSED ] 11 tests. 2024-03-26T04:50:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi012-78552-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/345595402' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: pgmap v264: 364 pgs: 8 unknown, 64 creating+peering, 292 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi012-78552-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:17.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T04:50:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/345595402' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:50:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:50:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: pgmap v264: 364 pgs: 8 unknown, 64 creating+peering, 292 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:17 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi012-78552-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi012-78567-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T04:50:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/345595402' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:50:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:50:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: pgmap v264: 364 pgs: 8 unknown, 64 creating+peering, 292 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:17 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]': finished 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]: dispatch 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]: dispatch 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:18.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]': finished 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]': finished 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]: dispatch 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]: dispatch 2024-03-26T04:50:18.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:18.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:18.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:18.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]': finished 2024-03-26T04:50:18.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-21"}]': finished 2024-03-26T04:50:18.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi012-78640-3","pool2":"test-rados-api-smithi012-78640-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T04:50:18.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T04:50:18.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]: dispatch 2024-03-26T04:50:18.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]: dispatch 2024-03-26T04:50:18.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:18.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:18.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:18.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-21", "mode": "writeback"}]': finished 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[41141]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[41141]: pgmap v267: 260 pgs: 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[29863]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:19.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:19.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:19.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[29863]: pgmap v267: 260 pgs: 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:50:19.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:19.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:19 smithi186 ceph-mon[33438]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T04:50:19.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:19.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:19.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:19.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:19 smithi186 ceph-mon[33438]: pgmap v267: 260 pgs: 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:50:19.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]': finished 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T04:50:20.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3790448378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]': finished 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T04:50:20.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3790448378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:20.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:20.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:20.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T04:50:20.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]: dispatch 2024-03-26T04:50:20.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]: dispatch 2024-03-26T04:50:20.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:20.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2833929654' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi012-78552-32"}]': finished 2024-03-26T04:50:20.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-21"}]': finished 2024-03-26T04:50:20.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T04:50:20.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3790448378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: pgmap v270: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3790448378' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:22.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: pgmap v270: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3790448378' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:22.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T04:50:22.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:50:22.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: pgmap v270: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3790448378' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi012-78552-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T04:50:22.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:23.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[29863]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2229087183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[41141]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2229087183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:23.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-mon[33438]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T04:50:23.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2229087183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:23.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: pgmap v273: 292 pgs: 64 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:23 smithi186 ceph-mon[33438]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T04:50:24.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: pgmap v273: 292 pgs: 64 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[29863]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: pgmap v273: 292 pgs: 64 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:24.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:24.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi012-78552-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:24.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:24.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:24.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:23 smithi012 ceph-mon[41141]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T04:50:25.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[29863]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T04:50:25.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2785610651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[29863]: pgmap v276: 332 pgs: 104 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[41141]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2785610651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:25.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:25 smithi012 ceph-mon[41141]: pgmap v276: 332 pgs: 104 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:25.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:25 smithi186 ceph-mon[33438]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T04:50:25.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2785610651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:25.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:25.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:25.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:25 smithi186 ceph-mon[33438]: pgmap v276: 332 pgs: 104 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2785610651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:26.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:50:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2785610651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:26.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:26.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2785610651' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:26.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:26.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T04:50:26.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:26.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:26.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:26.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]': finished 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]: dispatch 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]: dispatch 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/816538125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: pgmap v279: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]': finished 2024-03-26T04:50:27.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/816538125' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[29863]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]': finished 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]: dispatch 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]: dispatch 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/816538125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:27.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: pgmap v279: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T04:50:27.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:27.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]': finished 2024-03-26T04:50:27.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:27.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/816538125' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:27.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:27 smithi012 ceph-mon[41141]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-23"}]': finished 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1060876372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]: dispatch 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]: dispatch 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]: dispatch 2024-03-26T04:50:27.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/816538125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: pgmap v279: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T04:50:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-23", "mode": "writeback"}]': finished 2024-03-26T04:50:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi012-78552-33"}]': finished 2024-03-26T04:50:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/816538125' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:27.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:27 smithi186 ceph-mon[33438]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T04:50:28.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:28.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:28.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi012-78552-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:28.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi012-78552-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:28.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:28 smithi186 ceph-mon[33438]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T04:50:28.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi012-78552-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi012-78552-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi012-78552-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[29863]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi012-78552-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi012-78552-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:29.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi012-78552-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:29.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[41141]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T04:50:29.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi012-78552-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:30.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2349746820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: pgmap v282: 388 pgs: 64 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T04:50:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:30.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2349746820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: pgmap v282: 388 pgs: 64 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:30.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2349746820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:30.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:30.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:30.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:30.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: pgmap v282: 388 pgs: 64 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T04:50:30.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi012-78567-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:30.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:31.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:31.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T04:50:31.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:31.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:31.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi012-78552-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:31.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]': finished 2024-03-26T04:50:31.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:30 smithi186 ceph-mon[33438]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi012-78552-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]': finished 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[29863]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T04:50:31.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:31.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T04:50:31.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]: dispatch 2024-03-26T04:50:31.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:31.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi012-78552-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:31.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-23"}]': finished 2024-03-26T04:50:31.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:30 smithi012 ceph-mon[41141]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T04:50:32.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:31 smithi186 ceph-mon[33438]: pgmap v285: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:32.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:31 smithi186 ceph-mon[33438]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T04:50:32.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:50:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:50:32.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:31 smithi012 ceph-mon[29863]: pgmap v285: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:32.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:31 smithi012 ceph-mon[29863]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T04:50:32.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:31 smithi012 ceph-mon[41141]: pgmap v285: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:32.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:31 smithi012 ceph-mon[41141]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T04:50:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:32 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:32 smithi186 ceph-mon[33438]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T04:50:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[29863]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[41141]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:33.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:33.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:34.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:34.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: pgmap v288: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:50:34.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:34.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:34.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:34.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T04:50:34.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:34.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: pgmap v288: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: pgmap v288: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:50:34.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:34.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:34.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:34.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T04:50:34.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]: dispatch 2024-03-26T04:50:35.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:35.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T04:50:35.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:35.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:35.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: pgmap v291: 228 pgs: 32 unknown, 196 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:35.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:36.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: pgmap v291: 228 pgs: 32 unknown, 196 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:36.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/70759111' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi012-78552-34"}]': finished 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: pgmap v291: 228 pgs: 32 unknown, 196 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:36.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:36.932 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:50:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:37.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:37.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:37.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T04:50:37.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:37.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:37.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4065524092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:37.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:37.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:37.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4065524092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:37.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi012-78552-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4065524092' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:37.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]': finished 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]: dispatch 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: pgmap v294: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]: dispatch 2024-03-26T04:50:38.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:38.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:38.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:38.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]': finished 2024-03-26T04:50:38.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T04:50:38.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:38.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]': finished 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]: dispatch 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: pgmap v294: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]: dispatch 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]': finished 2024-03-26T04:50:38.406 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-25"}]': finished 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi012-78567-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]: dispatch 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: pgmap v294: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]: dispatch 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi012-78552-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:38.407 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-25", "mode": "writeback"}]': finished 2024-03-26T04:50:38.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T04:50:38.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:38.408 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:39.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[29863]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4074164186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi012-78567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[29863]: pgmap v297: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[41141]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4074164186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi012-78567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:39.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:39.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:39 smithi012 ceph-mon[41141]: pgmap v297: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:50:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:39 smithi186 ceph-mon[33438]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T04:50:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4074164186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi012-78567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]: dispatch 2024-03-26T04:50:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:39 smithi186 ceph-mon[33438]: pgmap v297: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:50:40.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4074164186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi012-78567-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]': finished 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[29863]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4074164186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi012-78567-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]': finished 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[41141]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:40.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:40.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:40 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:40.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4074164186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi012-78567-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:40.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-25"}]': finished 2024-03-26T04:50:40.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:40 smithi186 ceph-mon[33438]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T04:50:40.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:40.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: pgmap v300: 196 pgs: 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: pgmap v300: 196 pgs: 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T04:50:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793811392' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]: dispatch 2024-03-26T04:50:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: pgmap v300: 196 pgs: 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi012-78552-35"}]': finished 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:41.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:42.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:50:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:50:43.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:43.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi012-78552-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi012-78552-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:43.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:43.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:43.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:43.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:43.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi012-78552-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:43.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi012-78552-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[29863]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi012-78552-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[29863]: pgmap v303: 228 pgs: 32 unknown, 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi012-78552-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[41141]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T04:50:44.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi012-78552-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:44.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[41141]: pgmap v303: 228 pgs: 32 unknown, 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:44.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:44.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi012-78552-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:44.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:43 smithi186 ceph-mon[33438]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T04:50:44.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi012-78552-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:44.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:43 smithi186 ceph-mon[33438]: pgmap v303: 228 pgs: 32 unknown, 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:44.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:45.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi012-78552-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:45.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi012-78552-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:45.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:45.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:45.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T04:50:45.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi012-78552-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T04:50:45.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: pgmap v306: 244 pgs: 48 unknown, 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]': finished 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]: dispatch 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]: dispatch 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: pgmap v306: 244 pgs: 48 unknown, 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]': finished 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]: dispatch 2024-03-26T04:50:46.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]: dispatch 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: pgmap v306: 244 pgs: 48 unknown, 196 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-27"}]': finished 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T04:50:46.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:46.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:46.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]: dispatch 2024-03-26T04:50:46.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]: dispatch 2024-03-26T04:50:47.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:50:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]': finished 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:47.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]': finished 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:47.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:47.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-27", "mode": "writeback"}]': finished 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1353472116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:47.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]: dispatch 2024-03-26T04:50:48.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: pgmap v309: 228 pgs: 228 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:48.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: pgmap v309: 228 pgs: 228 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:48.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:48.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: pgmap v309: 228 pgs: 228 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-46"}]': finished 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]: dispatch 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:48.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:50:49.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:49.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:49.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:49.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:49.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:49.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:49 smithi186 ceph-mon[33438]: pgmap v312: 228 pgs: 228 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:49.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[29863]: pgmap v312: 228 pgs: 228 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:49.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3793366951' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi012-78552-36"}]': finished 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi012-78567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:49.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:49 smithi012 ceph-mon[41141]: pgmap v312: 228 pgs: 228 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:50:50.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:50 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:50.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]': finished 2024-03-26T04:50:50.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:50.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:50.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:50 smithi186 ceph-mon[33438]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T04:50:50.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:50.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:50.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]': finished 2024-03-26T04:50:50.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[29863]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-27"}]': finished 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[41141]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T04:50:50.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:51 smithi012 ceph-mon[29863]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T04:50:51.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:51 smithi012 ceph-mon[29863]: pgmap v315: 204 pgs: 8 unknown, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:51 smithi012 ceph-mon[41141]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T04:50:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:51 smithi012 ceph-mon[41141]: pgmap v315: 204 pgs: 8 unknown, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:51.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi012-78567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:51.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:51 smithi186 ceph-mon[33438]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T04:50:51.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:51 smithi186 ceph-mon[33438]: pgmap v315: 204 pgs: 8 unknown, 196 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:50:52.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:50:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:50:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[29863]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[41141]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:52.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:52 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:52.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi012-78552-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:52.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:52 smithi186 ceph-mon[33438]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T04:50:52.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:52.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:50:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[29863]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[29863]: pgmap v318: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[41141]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T04:50:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:53.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:53 smithi012 ceph-mon[41141]: pgmap v318: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:50:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:53 smithi186 ceph-mon[33438]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T04:50:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:53 smithi186 ceph-mon[33438]: pgmap v318: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[29863]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[41141]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:54.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2986396623' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:54 smithi186 ceph-mon[33438]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T04:50:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]: dispatch 2024-03-26T04:50:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[29863]: pgmap v321: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:55.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:55 smithi012 ceph-mon[41141]: pgmap v321: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi012-78567-47"}]': finished 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3492405244' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]: dispatch 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:50:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:55.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:55.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:55.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:55 smithi186 ceph-mon[33438]: pgmap v321: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:57.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T04:50:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:50:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:50:57.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:57.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:57.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:57.156 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:50:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:50:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi012-78552-37"}]': finished 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi012-78567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:57.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:57.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:50:58.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]': finished 2024-03-26T04:50:58.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:58.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]: dispatch 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: pgmap v324: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]: dispatch 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]': finished 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[41141]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T04:50:58.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]': finished 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]: dispatch 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: pgmap v324: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]: dispatch 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]': finished 2024-03-26T04:50:58.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:57 smithi012 ceph-mon[29863]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T04:50:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-29"}]': finished 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi012-78552-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]: dispatch 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: pgmap v324: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]: dispatch 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi012-78567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-29", "mode": "writeback"}]': finished 2024-03-26T04:50:58.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:57 smithi186 ceph-mon[33438]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T04:50:59.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:50:59.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:59 smithi012 ceph-mon[29863]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T04:50:59.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:50:59 smithi012 ceph-mon[29863]: pgmap v327: 244 pgs: 16 unknown, 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T04:50:59.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:50:59.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:59 smithi012 ceph-mon[41141]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T04:50:59.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:50:59 smithi012 ceph-mon[41141]: pgmap v327: 244 pgs: 16 unknown, 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T04:50:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi012-78552-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:50:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:59 smithi186 ceph-mon[33438]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T04:50:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:50:59 smithi186 ceph-mon[33438]: pgmap v327: 244 pgs: 16 unknown, 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T04:51:00.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:00 smithi012 ceph-mon[29863]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T04:51:00.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:00.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:00.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:00 smithi012 ceph-mon[41141]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T04:51:00.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:00.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:00.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:00 smithi186 ceph-mon[33438]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T04:51:00.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:00.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[29863]: pgmap v330: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:51:01.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:01.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:01 smithi012 ceph-mon[41141]: pgmap v330: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3082594515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]: dispatch 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:01 smithi186 ceph-mon[33438]: pgmap v330: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:02.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:51:02.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:51:02.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:02.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:02.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi012-78567-48"}]': finished 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/259626935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]: dispatch 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]: dispatch 2024-03-26T04:51:02.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:02.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:03.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:03.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:03.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]': finished 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi012-78552-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: pgmap v333: 228 pgs: 228 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:03.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]': finished 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi012-78552-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: pgmap v333: 228 pgs: 228 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:03.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-29"}]': finished 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi012-78552-38"}]': finished 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi012-78567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:03.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi012-78552-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: pgmap v333: 228 pgs: 228 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T04:51:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:03.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:05.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi012-78552-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:05.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:04 smithi012 ceph-mon[41141]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T04:51:05.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi012-78552-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi012-78552-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:04 smithi012 ceph-mon[29863]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T04:51:05.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi012-78552-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi012-78552-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:05.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:04 smithi186 ceph-mon[33438]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T04:51:05.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi012-78552-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:06.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: pgmap v336: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi012-78552-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:06.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: pgmap v336: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi012-78552-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:06.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi012-78567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: pgmap v336: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi012-78552-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:06.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:07.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:51:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:07.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:07.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:07.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:07.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T04:51:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:08.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: pgmap v339: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]': finished 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]: dispatch 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]: dispatch 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:08.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: pgmap v339: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]': finished 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]: dispatch 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]: dispatch 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:08.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: pgmap v339: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-31"}]': finished 2024-03-26T04:51:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T04:51:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]: dispatch 2024-03-26T04:51:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]: dispatch 2024-03-26T04:51:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1682494402' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:08.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]: dispatch 2024-03-26T04:51:09.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:09.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]': finished 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:09.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]': finished 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:09.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:09.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-31", "mode": "writeback"}]': finished 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi012-78567-49"}]': finished 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:09.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:09.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: pgmap v342: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: pgmap v342: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:10.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:10.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:10.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: pgmap v342: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4291826884' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi012-78552-39"}]': finished 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi012-78567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]: dispatch 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:10.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:10.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]': finished 2024-03-26T04:51:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[29863]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]': finished 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[41141]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T04:51:11.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:10 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-31"}]': finished 2024-03-26T04:51:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi012-78552-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:11.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:10 smithi186 ceph-mon[33438]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T04:51:11.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:12.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:12 smithi012 ceph-mon[29863]: pgmap v345: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:12.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:12.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:12 smithi012 ceph-mon[29863]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T04:51:12.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:12 smithi012 ceph-mon[41141]: pgmap v345: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:12.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:12.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:12 smithi012 ceph-mon[41141]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T04:51:12.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:51:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:51:12.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:12 smithi186 ceph-mon[33438]: pgmap v345: 228 pgs: 228 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:12.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi012-78567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:12.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:12 smithi186 ceph-mon[33438]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T04:51:13.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:13.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:13.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[41141]: pgmap v348: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:13.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:13 smithi012 ceph-mon[29863]: pgmap v348: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:13.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:13.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:13.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi012-78552-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:13.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T04:51:13.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:13.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:13.461 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:13 smithi186 ceph-mon[33438]: pgmap v348: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:14.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:14.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[29863]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[41141]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:14.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:14.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:14.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:14 smithi186 ceph-mon[33438]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T04:51:14.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:14.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:14.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:14.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:15.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:15.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:15.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[29863]: pgmap v351: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:15.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T04:51:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:15.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:15 smithi012 ceph-mon[41141]: pgmap v351: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:15.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1569322313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]: dispatch 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:15.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:15 smithi186 ceph-mon[33438]: pgmap v351: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:16.902 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:51:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:16.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]': finished 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]: dispatch 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]: dispatch 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]': finished 2024-03-26T04:51:16.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]: dispatch 2024-03-26T04:51:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T04:51:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]: dispatch 2024-03-26T04:51:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:16.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:16.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi012-78567-50"}]': finished 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-33"}]': finished 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]: dispatch 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]: dispatch 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3500376690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:16.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]: dispatch 2024-03-26T04:51:17.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]': finished 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1450771490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:17.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: pgmap v354: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1450771490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:17.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]': finished 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1450771490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: pgmap v354: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1450771490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:17.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-33", "mode": "writeback"}]': finished 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi012-78552-40"}]': finished 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1450771490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:17.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: pgmap v354: 260 pgs: 32 unknown, 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1450771490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi012-78567-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi012-78552-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]: dispatch 2024-03-26T04:51:17.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]': finished 2024-03-26T04:51:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]': finished 2024-03-26T04:51:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:18 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:19.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:18 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:19.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:18 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:19.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:18 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:19.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-33"}]': finished 2024-03-26T04:51:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T04:51:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: pgmap v357: 228 pgs: 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: pgmap v357: 228 pgs: 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: pgmap v357: 228 pgs: 228 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi012-78552-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T04:51:20.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:20.212 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[29863]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T04:51:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[29863]: pgmap v360: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:21.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:21.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[41141]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T04:51:21.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:21.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[41141]: pgmap v360: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:21.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:21 smithi186 ceph-mon[33438]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T04:51:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:21 smithi186 ceph-mon[33438]: pgmap v360: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:21.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:22.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:51:22.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:22.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:22.906 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:22.907 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi012-78567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:22.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:23.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:23.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:23.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: pgmap v363: 236 pgs: 32 creating+peering, 8 unknown, 196 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]: dispatch 2024-03-26T04:51:23.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]: dispatch 2024-03-26T04:51:23.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:23.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:23.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:23.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T04:51:23.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:23.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: pgmap v363: 236 pgs: 32 creating+peering, 8 unknown, 196 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]: dispatch 2024-03-26T04:51:23.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]: dispatch 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1021478735' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]: dispatch 2024-03-26T04:51:23.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: pgmap v363: 236 pgs: 32 creating+peering, 8 unknown, 196 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:23.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:23.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi012-78552-41"}]': finished 2024-03-26T04:51:23.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T04:51:23.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]: dispatch 2024-03-26T04:51:23.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]: dispatch 2024-03-26T04:51:24.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:24.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]': finished 2024-03-26T04:51:24.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]': finished 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:24.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-35", "mode": "writeback"}]': finished 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi012-78552-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2473733332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]: dispatch 2024-03-26T04:51:24.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:26.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:25 smithi012 ceph-mon[29863]: pgmap v366: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:26.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:25 smithi012 ceph-mon[41141]: pgmap v366: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:25 smithi186 ceph-mon[33438]: pgmap v366: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:51:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi012-78567-52"}]': finished 2024-03-26T04:51:27.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:51:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:27.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:27.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:27.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:27.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi012-78552-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:27.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:28.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:27 smithi012 ceph-mon[29863]: pgmap v369: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:51:28.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:28.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:27 smithi012 ceph-mon[41141]: pgmap v369: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:51:28.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:28.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:27 smithi186 ceph-mon[33438]: pgmap v369: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:51:28.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T04:51:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:29.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:29.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:29.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T04:51:29.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:29.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi012-78567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:29.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:29.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T04:51:29.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:29.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: pgmap v372: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:51:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T04:51:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: pgmap v372: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:30.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:30.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: pgmap v372: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:51:30.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:30.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T04:51:30.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3079784556' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]: dispatch 2024-03-26T04:51:30.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:30.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7049 ms) 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6892 ms) 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7003 ms) 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-26T04:51:30.639 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6993 ms) 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7033 ms) 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7035 ms) 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7021 ms) 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7034 ms) 2024-03-26T04:51:30.640 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6889 ms) 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7017 ms) 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7062 ms) 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6944 ms) 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6979 ms) 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7042 ms) 2024-03-26T04:51:30.641 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7019 ms) 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7017 ms) 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112030 ms total) 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [----------] Global test environment tear-down 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (213268 ms total) 2024-03-26T04:51:30.642 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-26T04:51:31.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:31.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:31.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T04:51:31.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[29863]: pgmap v375: 228 pgs: 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:31.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:31.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:31.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:31.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:31 smithi012 ceph-mon[41141]: pgmap v375: 228 pgs: 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:31.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi012-78552-42"}]': finished 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]: dispatch 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:31.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:31 smithi186 ceph-mon[33438]: pgmap v375: 228 pgs: 228 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:32.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:51:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T04:51:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:32.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:32.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:32.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:32.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-35"}]': finished 2024-03-26T04:51:32.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:32.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T04:51:32.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2755238676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:32.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]: dispatch 2024-03-26T04:51:33.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:33.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:33.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T04:51:33.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: pgmap v378: 196 pgs: 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T04:51:33.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: pgmap v378: 196 pgs: 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:33.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi012-78567-53"}]': finished 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: pgmap v378: 196 pgs: 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:33.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:35.213 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:35.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:34 smithi186 ceph-mon[33438]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T04:51:35.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:35.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:35.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:35.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[29863]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T04:51:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi012-78567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[41141]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T04:51:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:36.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:36.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:36.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: pgmap v381: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:36.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: pgmap v381: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: pgmap v381: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T04:51:36.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:36.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi012-78567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:36.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:36.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T04:51:36.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:36.404 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:37.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:51:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:51:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:37.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]': finished 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]: dispatch 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]: dispatch 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: pgmap v384: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]': finished 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:37.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]': finished 2024-03-26T04:51:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]: dispatch 2024-03-26T04:51:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T04:51:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]: dispatch 2024-03-26T04:51:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: pgmap v384: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]': finished 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-37"}]': finished 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]: dispatch 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]: dispatch 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: pgmap v384: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:38.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:38.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-37", "mode": "writeback"}]': finished 2024-03-26T04:51:38.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T04:51:38.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:38.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:38.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:38.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:38.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:38.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:38.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:38.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:38.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T04:51:38.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:38.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:38.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]: dispatch 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1397556213' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]: dispatch 2024-03-26T04:51:39.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: pgmap v387: 228 pgs: 228 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:39.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:39.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]': finished 2024-03-26T04:51:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T04:51:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:39.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi012-78567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: pgmap v387: 228 pgs: 228 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]': finished 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi012-78567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: pgmap v387: 228 pgs: 228 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-37"}]': finished 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi012-78567-54"}]': finished 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi012-78567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:41.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi012-78567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:41.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[29863]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T04:51:41.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi012-78567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:41.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[29863]: pgmap v390: 196 pgs: 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:41.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi012-78567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:41.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[41141]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T04:51:41.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi012-78567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:41.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:41 smithi012 ceph-mon[41141]: pgmap v390: 196 pgs: 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:41.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi012-78567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:41.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:41 smithi186 ceph-mon[33438]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T04:51:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi012-78567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:41.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:41 smithi186 ceph-mon[33438]: pgmap v390: 196 pgs: 196 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:42.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:51:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:51:42.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:42 smithi012 ceph-mon[29863]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T04:51:42.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:42.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:42.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:42 smithi012 ceph-mon[41141]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T04:51:42.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:42.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:42.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:42 smithi186 ceph-mon[33438]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T04:51:42.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:42.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi012-78567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[29863]: pgmap v393: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:51:43.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:43.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi012-78567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:43.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:43.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T04:51:43.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:43.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:43.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:43 smithi012 ceph-mon[41141]: pgmap v393: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:51:43.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:43.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi012-78567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:43.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:43.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T04:51:43.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:43.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:43.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:43 smithi186 ceph-mon[33438]: pgmap v393: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:51:44.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:44.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:44.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:44.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[29863]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T04:51:44.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:44.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:44.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:44.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:44.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[41141]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T04:51:44.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:44.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:44 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:44.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:44.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:44.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:44 smithi186 ceph-mon[33438]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T04:51:44.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:45.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]': finished 2024-03-26T04:51:45.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T04:51:45.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]: dispatch 2024-03-26T04:51:45.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]: dispatch 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: pgmap v396: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]': finished 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]': finished 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T04:51:45.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]: dispatch 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]: dispatch 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: pgmap v396: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]': finished 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T04:51:45.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:45.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-39"}]': finished 2024-03-26T04:51:45.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T04:51:45.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]: dispatch 2024-03-26T04:51:45.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]: dispatch 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: pgmap v396: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-39", "mode": "writeback"}]': finished 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T04:51:45.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]: dispatch 2024-03-26T04:51:46.902 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:51:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:46.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:46.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:46.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:51:46.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:46.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:46.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:46.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:46.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:46.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:51:46.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1659181843' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi012-78567-55"}]': finished 2024-03-26T04:51:46.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:51:46.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T04:51:46.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:46.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]: dispatch 2024-03-26T04:51:47.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: pgmap v399: 228 pgs: 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]': finished 2024-03-26T04:51:47.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: pgmap v399: 228 pgs: 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]': finished 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.905 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: pgmap v399: 228 pgs: 228 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-39"}]': finished 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi012-78567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:47.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T04:51:47.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:47.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: seed 78824 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (387 ms) 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (20359 ms) 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (111 ms) 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (20857 ms total) 2024-03-26T04:51:48.841 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (10212 ms) 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7998 ms) 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10018 ms) 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: my_snaps [3] 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: my_snaps [4,3] 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: promoting some heads 2024-03-26T04:51:48.842 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: done waiting 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46227 ms) 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9761 ms) 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8006 ms) 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8028 ms) 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8040 ms) 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-26T04:51:48.843 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10045 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8892 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10284 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13006 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7977 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7783 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13044 ms) 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-26T04:51:48.844 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7913 ms) 2024-03-26T04:51:48.845 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-26T04:51:48.845 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8285 ms) 2024-03-26T04:51:49.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:49.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:49.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[29863]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[29863]: pgmap v402: 196 pgs: 196 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[29863]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:50.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[41141]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T04:51:50.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[41141]: pgmap v402: 196 pgs: 196 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T04:51:50.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:50.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[41141]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T04:51:50.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:50.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:50.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:49 smithi186 ceph-mon[33438]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T04:51:50.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:49 smithi186 ceph-mon[33438]: pgmap v402: 196 pgs: 196 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T04:51:50.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi012-78567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:50.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:49 smithi186 ceph-mon[33438]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T04:51:50.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:50.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[29863]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T04:51:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:51.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[29863]: pgmap v405: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:51.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[41141]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T04:51:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:51.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:51 smithi012 ceph-mon[41141]: pgmap v405: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:51.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:51.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:51 smithi186 ceph-mon[33438]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T04:51:51.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:51.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:51.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:51 smithi186 ceph-mon[33438]: pgmap v405: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:51:52.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:52.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:52.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:52.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:51:52.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:51:52.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T04:51:52.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:51:52.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:52.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[29863]: pgmap v408: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:53 smithi012 ceph-mon[41141]: pgmap v408: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:53.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:53.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:51:53.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:53.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:51:53.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T04:51:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:51:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3557830195' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]: dispatch 2024-03-26T04:51:53.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:53 smithi186 ceph-mon[33438]: pgmap v408: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:51:54.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:54.904 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:54 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:51:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi012-78567-56"}]': finished 2024-03-26T04:51:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T04:51:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:51:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:51:54.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:54.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:54 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[29863]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[29863]: pgmap v411: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:51:55.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:55.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[41141]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T04:51:55.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:55.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:55.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:55 smithi012 ceph-mon[41141]: pgmap v411: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:55.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:51:55.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi012-78567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:51:55.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:55 smithi186 ceph-mon[33438]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T04:51:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:55.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:55 smithi186 ceph-mon[33438]: pgmap v411: 228 pgs: 228 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:56.902 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:51:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:56.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[29863]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T04:51:56.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:56.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:56.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]: dispatch 2024-03-26T04:51:56.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]: dispatch 2024-03-26T04:51:56.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[41141]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T04:51:56.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:56.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:56.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]: dispatch 2024-03-26T04:51:56.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]: dispatch 2024-03-26T04:51:56.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:56 smithi186 ceph-mon[33438]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T04:51:56.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:56.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:51:56.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]: dispatch 2024-03-26T04:51:56.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]: dispatch 2024-03-26T04:51:57.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:51:57.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]': finished 2024-03-26T04:51:57.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[41141]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T04:51:57.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[41141]: pgmap v414: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:57.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:57.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:51:57.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]': finished 2024-03-26T04:51:57.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[29863]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T04:51:57.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[29863]: pgmap v414: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:57.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:57 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:57.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi012-78567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:51:57.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-41"}]': finished 2024-03-26T04:51:57.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:57 smithi186 ceph-mon[33438]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T04:51:57.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:57 smithi186 ceph-mon[33438]: pgmap v414: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:57.963 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:57 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:58.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:58 smithi186 ceph-mon[33438]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T04:51:59.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:58 smithi012 ceph-mon[29863]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T04:51:59.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:58 smithi012 ceph-mon[41141]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T04:51:59.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T04:51:59.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:59.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:59.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:59.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:51:59.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: pgmap v417: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi012-78824-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:51:59.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:51:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: pgmap v417: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi012-78824-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: pgmap v417: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi012-78824-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:52:00.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:00.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:00.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2005392097' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:00.155 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:51:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]: dispatch 2024-03-26T04:52:01.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:01.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:52:01.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T04:52:01.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:52:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:52:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi012-78567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:01.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:01 smithi186 ceph-mon[33438]: pgmap v420: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi012-78567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:02.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[29863]: pgmap v420: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi012-78567-57"}]': finished 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi012-78567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:02.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:01 smithi012 ceph-mon[41141]: pgmap v420: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:02.462 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:52:03.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T04:52:03.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi012-78567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:03.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:03.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-mon[33438]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T04:52:03.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi012-78567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:03.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi012-78567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[29863]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi012-78567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T04:52:03.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi012-78567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:03.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:03.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[41141]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T04:52:03.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi012-78567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:03.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: ok, hit_set contains 276:602f83fe:::foo:head 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8806 ms) 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg_num = 32 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 0 ls 1711428724,0 2024-03-26T04:52:03.902 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 1 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 2 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 3 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 4 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 5 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 6 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 7 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 8 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 9 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 10 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 11 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 12 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 13 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 14 ls 1711428724,0 2024-03-26T04:52:03.903 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 15 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 16 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 17 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 18 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 19 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 20 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 21 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 22 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 23 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 24 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 25 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 26 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 27 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 28 ls 1711428724,0 2024-03-26T04:52:03.904 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 29 ls 1711428724,0 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 30 ls 1711428724,0 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg 31 ls 1711428724,0 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: pg_num = 32 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6cac518f:::0:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:02547ec2:::1:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f905c69b:::2:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:cfc208b3:::3:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d83876eb:::4:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b29083e3:::5:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c4fdafeb:::6:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:5c6b0b28:::7:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:bd63b0f1:::8:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e960b815:::9:head 2024-03-26T04:52:03.905 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:52ea6a34:::10:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:89d3ae78:::11:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:de5d7c5f:::12:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:566253c9:::13:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:62a1935d:::14:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:863748b0:::15:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3958e169:::16:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4d4dabf9:::17:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:8391935d:::18:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:28883081:::19:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:69259c59:::20:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4bdb80b7:::21:head 2024-03-26T04:52:03.906 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a11c5d71:::22:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:271af37b:::23:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:95b121be:::24:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:58d1031b:::25:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:0a050783:::26:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c709704c:::27:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:cbe56eaf:::28:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:86b4b162:::29:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:70d89383:::30:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:dd450c7c:::31:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6d5729b1:::32:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c388f3fb:::33:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:56cfea31:::34:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9dbc1bf7:::35:head 2024-03-26T04:52:03.907 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:40b74ccd:::36:head 2024-03-26T04:52:03.908 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4d5aaf42:::37:head 2024-03-26T04:52:03.908 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:920f362c:::38:head 2024-03-26T04:52:03.908 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6cc53222:::39:head 2024-03-26T04:52:03.908 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9cad833f:::40:head 2024-03-26T04:52:03.908 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:1ea84d41:::41:head 2024-03-26T04:52:03.908 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c4480ef6:::42:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a694361e:::43:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d1bd33e9:::44:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ddc2cd5d:::45:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:2b782207:::46:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7b187fca:::47:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:90ecdf6f:::48:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a5ed95fe:::49:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ea0eaa55:::50:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f33ef17b:::51:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a0d1b2f6:::52:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:60c5229e:::53:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:edcbc575:::54:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:102cf253:::55:head 2024-03-26T04:52:03.909 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:efb7fb0b:::56:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:50d0a326:::57:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d4dc5daf:::58:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3a130462:::59:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ec87ed71:::60:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d5bc9454:::61:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3ddfe313:::62:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7c2816b9:::63:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:47e00e4d:::64:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c6410c18:::65:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b48ed237:::66:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:cd63ad31:::67:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b179e92b:::68:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:0d9f741a:::69:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6d3352ae:::70:head 2024-03-26T04:52:03.910 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c6d5c19e:::71:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:bc4729c3:::72:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:77e930b9:::73:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:0abeecfd:::74:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b7c37e15:::75:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b6378398:::76:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:02bd68de:::77:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:cc795d2d:::78:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:630d4fea:::79:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e0d29ef5:::80:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:fd6f13d2:::81:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:606461d5:::82:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:eadbdc43:::83:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:8761d0bb:::84:head 2024-03-26T04:52:03.911 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9ef0186f:::85:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e0d41294:::86:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:961de695:::87:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:1423148f:::88:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:633a8fa2:::89:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a8653809:::90:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3dac8b33:::91:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:35aad435:::92:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f6dcc343:::93:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:dbbdad87:::94:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:1cb48ce0:::95:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:03cd461c:::96:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:17a4ea99:::97:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9993c9a7:::98:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6394211c:::99:head 2024-03-26T04:52:03.912 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:94c7ae57:::100:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6fdee5bb:::101:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9a477fd1:::102:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:eb850916:::103:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:affc56b9:::104:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b42dc814:::105:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f319f8f0:::106:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9a40b9de:::107:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:8b524f28:::108:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e3de589f:::109:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:90f90a5b:::110:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a7b4f1d7:::111:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:af51766e:::112:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b6f90bd1:::113:head 2024-03-26T04:52:03.913 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e0261208:::114:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c9569ef7:::115:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:61bebe50:::116:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:fe93412b:::117:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d3d38bee:::118:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3100ba0c:::119:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d0560ada:::120:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f0ea8b35:::121:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:766f231a:::122:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a07a2582:::123:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:bd7c6b3a:::124:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:fb2ddaff:::125:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4408e1fe:::126:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ee1df7a7:::127:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c3002909:::128:head 2024-03-26T04:52:03.914 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4f48ffa9:::129:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:edf38733:::130:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c08425c0:::131:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:5f902d98:::132:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:41ea2c93:::133:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:813cee13:::134:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:0131818d:::135:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:26ba5a85:::136:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:381b8a5a:::137:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:28797e47:::138:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:bfca7f22:::139:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:36807075:::140:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:80b03975:::141:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:5c15709b:::142:head 2024-03-26T04:52:03.915 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f39ea15e:::143:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ea992956:::144:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:48887b1c:::145:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:9f24a9dd:::146:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:987f100b:::147:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d2dd3581:::148:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7fed1808:::149:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c80b70e9:::150:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:85ed90f9:::151:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:36428b24:::152:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d044c34a:::153:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7c18bf58:::154:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d1c21232:::155:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a7a3c575:::156:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:87da0633:::157:head 2024-03-26T04:52:03.916 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d5ac3822:::158:head 2024-03-26T04:52:03.917 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3f20522d:::159:head 2024-03-26T04:52:03.917 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6ca26563:::160:head 2024-03-26T04:52:03.917 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:532ce135:::161:head 2024-03-26T04:52:03.917 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:c78863e6:::162:head 2024-03-26T04:52:04.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:04.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:52:04.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:52:04.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: pgmap v423: 228 pgs: 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:04.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T04:52:04.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:52:04.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:04.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi012-78567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:04.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T04:52:04.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T04:52:04.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi012-78824-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: pgmap v423: 228 pgs: 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi012-78567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T04:52:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi012-78824-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: pgmap v423: 228 pgs: 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi012-78567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T04:52:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi012-78824-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:52:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]: dispatch 2024-03-26T04:52:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]: dispatch 2024-03-26T04:52:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]': finished 2024-03-26T04:52:05.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:04 smithi186 ceph-mon[33438]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]: dispatch 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]: dispatch 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]': finished 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[29863]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]: dispatch 2024-03-26T04:52:05.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]: dispatch 2024-03-26T04:52:05.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-43"}]': finished 2024-03-26T04:52:05.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:04 smithi012 ceph-mon[41141]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T04:52:06.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:05 smithi186 ceph-mon[33438]: pgmap v426: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:06.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:05 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:06.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:05 smithi186 ceph-mon[33438]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T04:52:06.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:05 smithi012 ceph-mon[29863]: pgmap v426: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:06.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:05 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:06.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:05 smithi012 ceph-mon[29863]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T04:52:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:05 smithi012 ceph-mon[41141]: pgmap v426: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:05 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:05 smithi012 ceph-mon[41141]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T04:52:06.942 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:52:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:07.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:07.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:07.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:06 smithi186 ceph-mon[33438]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T04:52:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:07.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:07.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:07.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:07.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[29863]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T04:52:07.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:07.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:07.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[41141]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T04:52:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]: dispatch 2024-03-26T04:52:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:07.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:08.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: pgmap v429: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:08.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:08.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:08.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T04:52:08.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:08.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:08.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: pgmap v429: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: pgmap v429: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3784884592' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi012-78567-58"}]': finished 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:08.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:08.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:09.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:09.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:09.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T04:52:09.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:52:09.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:52:09.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:09.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:09.961 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:09 smithi186 ceph-mon[33438]: pgmap v432: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:10.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[41141]: pgmap v432: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:10.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:09 smithi012 ceph-mon[29863]: pgmap v432: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:10.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:52:10.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:52:10.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:10 smithi186 ceph-mon[33438]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T04:52:10.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[29863]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[41141]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T04:52:11.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:52:11.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:11.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:52:11.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:11 smithi186 ceph-mon[33438]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T04:52:11.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:11.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:11 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:11.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:11 smithi186 ceph-mon[33438]: pgmap v435: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[29863]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[29863]: pgmap v435: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi012-78567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[41141]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T04:52:12.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:12.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:12.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:11 smithi012 ceph-mon[41141]: pgmap v435: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:12.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:52:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[29863]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:52:13.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[41141]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T04:52:13.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:52:13.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:12 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:13.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:52:13.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:52:13.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:12 smithi186 ceph-mon[33438]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T04:52:13.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: pgmap v438: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[29863]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T04:52:14.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: pgmap v438: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:14.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:13 smithi012 ceph-mon[41141]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T04:52:14.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:14.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:52:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T04:52:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: pgmap v438: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:14.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:13 smithi186 ceph-mon[33438]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T04:52:14.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7462ddf6:o.RoundTripAppendPP (3005 ms) 2024-03-26T04:52:14.727 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3260 ms) 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2749 ms) 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3010 ms) 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: setting pool EIO 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: max_success 100, min_failed 101 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3985 ms) 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3009 ms) 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (130182 ms total) 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: 2024-03-26T04:52:14.728 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3012 ms) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9046 ms) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15059 ms) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2878 ms) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29995 ms total) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3007 ms) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3007 ms total) 2024-03-26T04:52:14.729 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14071 ms) 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6949 ms) 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6979 ms) 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7043 ms) 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3265 ms) 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6761 ms) 2024-03-26T04:52:14.730 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7029 ms) 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6911 ms) 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7026 ms) 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7031 ms) 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7042 ms) 2024-03-26T04:52:14.731 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-26T04:52:14.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7032 ms) 2024-03-26T04:52:14.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-26T04:52:14.732 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7038 ms) 2024-03-26T04:52:15.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:15.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:15.053 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:15.054 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:15.054 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:15.054 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:15.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/167298376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:15.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]: dispatch 2024-03-26T04:52:15.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi012-78567-59"}]': finished 2024-03-26T04:52:16.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:15 smithi012 ceph-mon[29863]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T04:52:16.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:15 smithi012 ceph-mon[29863]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:16.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:15 smithi012 ceph-mon[41141]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T04:52:16.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:15 smithi012 ceph-mon[41141]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:16.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:15 smithi186 ceph-mon[33438]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T04:52:16.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:15 smithi186 ceph-mon[33438]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:17.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-mon[29863]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T04:52:17.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/404102721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:17.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:17.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:52:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:17.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-mon[41141]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T04:52:17.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/404102721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:17.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:17.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:16 smithi186 ceph-mon[33438]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T04:52:17.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/404102721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:17.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[29863]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[29863]: pgmap v444: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[41141]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[41141]: pgmap v444: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:52:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:17 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:18.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi012-78567-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:18.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:17 smithi186 ceph-mon[33438]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T04:52:18.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:17 smithi186 ceph-mon[33438]: pgmap v444: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:52:18.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:17 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:19.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:19.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:18 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:19.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:19.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:18 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:20.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[29863]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[29863]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[41141]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:20.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:19 smithi012 ceph-mon[41141]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:52:20.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi012-78567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:20.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:19 smithi186 ceph-mon[33438]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T04:52:20.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:20.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:20.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:19 smithi186 ceph-mon[33438]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:52:21.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:20 smithi012 ceph-mon[29863]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T04:52:21.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:20 smithi012 ceph-mon[41141]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T04:52:21.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:20 smithi186 ceph-mon[33438]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T04:52:22.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[29863]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[29863]: pgmap v450: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[29863]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[41141]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[41141]: pgmap v450: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:22.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:21 smithi012 ceph-mon[41141]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T04:52:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi012-78567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:21 smithi186 ceph-mon[33438]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T04:52:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:21 smithi186 ceph-mon[33438]: pgmap v450: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:21 smithi186 ceph-mon[33438]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T04:52:22.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:52:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:52:23.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[29863]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[41141]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:23.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:23.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:22 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:23.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:22 smithi186 ceph-mon[33438]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T04:52:23.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:23.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[29863]: pgmap v453: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[29863]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[41141]: pgmap v453: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:24.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[41141]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T04:52:24.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:24.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:23 smithi186 ceph-mon[33438]: pgmap v453: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:52:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:23 smithi186 ceph-mon[33438]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T04:52:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3142536853' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:24.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]: dispatch 2024-03-26T04:52:25.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:25.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:25.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:25.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[29863]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T04:52:25.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]: dispatch 2024-03-26T04:52:25.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]: dispatch 2024-03-26T04:52:25.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:25.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:25.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:25.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[41141]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T04:52:25.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]: dispatch 2024-03-26T04:52:25.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]: dispatch 2024-03-26T04:52:25.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:25.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:25.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi012-78567-61"}]': finished 2024-03-26T04:52:25.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:24 smithi186 ceph-mon[33438]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T04:52:25.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]: dispatch 2024-03-26T04:52:25.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: pgmap v456: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]': finished 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T04:52:26.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: pgmap v456: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]': finished 2024-03-26T04:52:26.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:26.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T04:52:26.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:26.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: pgmap v456: 228 pgs: 228 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:52:26.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-45"}]': finished 2024-03-26T04:52:26.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:26.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T04:52:26.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:26.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:27.151 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:52:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:52:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:28.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:27 smithi012 ceph-mon[29863]: pgmap v459: 196 pgs: 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:28.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:27 smithi012 ceph-mon[29863]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T04:52:28.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:27 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:28.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:27 smithi012 ceph-mon[41141]: pgmap v459: 196 pgs: 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:28.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:27 smithi012 ceph-mon[41141]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T04:52:28.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:27 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:28.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:27 smithi186 ceph-mon[33438]: pgmap v459: 196 pgs: 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:28.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:27 smithi186 ceph-mon[33438]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T04:52:28.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:27 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:29.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[29863]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[41141]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:29.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:29.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi012-78567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:29.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:28 smithi186 ceph-mon[33438]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T04:52:29.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:29.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:30.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[29863]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[29863]: pgmap v462: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[41141]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:30.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:29 smithi012 ceph-mon[41141]: pgmap v462: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:30.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:30.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:29 smithi186 ceph-mon[33438]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T04:52:30.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:30.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:30.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:29 smithi186 ceph-mon[33438]: pgmap v462: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:31.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[29863]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:31.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[41141]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T04:52:31.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:31.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:31.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:31.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:31.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:31.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:30 smithi186 ceph-mon[33438]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T04:52:31.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:31.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:31.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]': finished 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]: dispatch 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]: dispatch 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: pgmap v465: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:52:32.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]': finished 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]: dispatch 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]: dispatch 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: pgmap v465: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:32.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:31 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-47"}]': finished 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]: dispatch 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/4052118586' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]: dispatch 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]: dispatch 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: pgmap v465: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:32.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:31 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:52:32.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:52:33.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]': finished 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi012-78567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi012-78567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:33.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi012-78567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]': finished 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi012-78567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:52:33.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi012-78567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:33.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T04:52:33.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:33.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi012-78567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.154 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-47", "mode": "writeback"}]': finished 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi012-78567-62"}]': finished 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi012-78567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi012-78567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi012-78567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:33.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:52:34.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[29863]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[41141]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:34.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:52:34.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:34 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:34.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:34 smithi186 ceph-mon[33438]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:52:34.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:34 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:52:34.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:34.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:52:35.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi012-78567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[29863]: pgmap v471: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:52:35.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi012-78567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:52:35.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:35 smithi012 ceph-mon[41141]: pgmap v471: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi012-78567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:52:35.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:35 smithi186 ceph-mon[33438]: pgmap v471: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:52:36.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[29863]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[41141]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:52:36.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:52:36.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:36 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:36.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:52:36.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:36 smithi186 ceph-mon[33438]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T04:52:36.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:52:36.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:52:37.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:52:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:52:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:52:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[29863]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[29863]: pgmap v474: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[41141]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:52:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:37 smithi012 ceph-mon[41141]: pgmap v474: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:38.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:52:38.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:37 smithi186 ceph-mon[33438]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T04:52:38.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:52:38.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:38.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:52:38.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:37 smithi186 ceph-mon[33438]: pgmap v474: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[29863]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[41141]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:39.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:39.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:39.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:39.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:52:39.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:38 smithi186 ceph-mon[33438]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T04:52:39.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]: dispatch 2024-03-26T04:52:39.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:39.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:40.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:40.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[29863]: pgmap v477: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:40.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:39 smithi012 ceph-mon[41141]: pgmap v477: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2600826471' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi012-78567-63"}]': finished 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:40.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:40.211 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:39 smithi186 ceph-mon[33438]: pgmap v477: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:41.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]': finished 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]': finished 2024-03-26T04:52:41.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:41.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]': finished 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi012-78567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:41.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-47"}]: dispatch 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[29863]: pgmap v480: 196 pgs: 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[29863]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[29863]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[41141]: pgmap v480: 196 pgs: 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[41141]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T04:52:42.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:42.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[41141]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T04:52:42.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:42.404 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:42.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:52:42.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-mon[33438]: pgmap v480: 196 pgs: 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:42.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-mon[33438]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T04:52:42.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi012-78567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:42.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-mon[33438]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T04:52:42.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:42.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:52:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: pgmap v483: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T04:52:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: pgmap v483: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:52:43.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:43.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:43.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:43.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:52:43.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T04:52:43.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: pgmap v483: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:52:43.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:43.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:52:45.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[29863]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[41141]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:45.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:45.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:52:45.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:45.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:44 smithi186 ceph-mon[33438]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T04:52:45.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:45.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:45.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]': finished 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: pgmap v486: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]': finished 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[29863]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T04:52:46.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]': finished 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: pgmap v486: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]': finished 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:46.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:45 smithi012 ceph-mon[41141]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T04:52:46.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-49"}]': finished 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/725858852' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: pgmap v486: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]: dispatch 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-49", "mode": "readproxy"}]': finished 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi012-78567-64"}]': finished 2024-03-26T04:52:46.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:45 smithi186 ceph-mon[33438]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T04:52:47.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:52:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:47.153 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:47.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:47.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:47.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:48.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:48 smithi186 ceph-mon[33438]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:52:48.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:48.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:48 smithi186 ceph-mon[33438]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T04:52:48.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:48.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:48.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:52:48.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[29863]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:52:48.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:48.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[29863]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T04:52:48.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[41141]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi012-78567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[41141]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:48.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:52:49.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: pgmap v491: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:49.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:49 smithi186 ceph-mon[33438]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T04:52:49.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T04:52:49.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:52:49.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:52:49.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: pgmap v491: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[29863]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: pgmap v491: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi012-78567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:49.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:49 smithi012 ceph-mon[41141]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T04:52:51.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:50 smithi186 ceph-mon[33438]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T04:52:51.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:50 smithi012 ceph-mon[29863]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T04:52:51.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:50 smithi012 ceph-mon[41141]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T04:52:52.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:51 smithi186 ceph-mon[33438]: pgmap v494: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:52.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:51 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:52.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:51 smithi186 ceph-mon[33438]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T04:52:52.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:52.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:51 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:52.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[41141]: pgmap v494: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:52.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:52.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[41141]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T04:52:52.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:52.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[29863]: pgmap v494: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:52:52.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:52.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[29863]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T04:52:52.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:52.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:51 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:52.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:52:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:52:53.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:53.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:52 smithi186 ceph-mon[33438]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T04:52:53.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:53.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:52 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:53.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:53.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[29863]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T04:52:53.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:53.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:53.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:53.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[41141]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T04:52:53.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2096276329' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:53.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:52 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]: dispatch 2024-03-26T04:52:54.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:54.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:54.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:54.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T04:52:54.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:54.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:54.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi012-78567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:54.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:54.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:54.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi012-78567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi012-78567-65"}]': finished 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:54.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:54.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi012-78567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:52:55.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi012-78567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:55.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:54 smithi186 ceph-mon[33438]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T04:52:55.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi012-78567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:55.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi012-78567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:55.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:54 smithi012 ceph-mon[29863]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T04:52:55.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi012-78567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:55.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi012-78567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:52:55.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:54 smithi012 ceph-mon[41141]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T04:52:55.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi012-78567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:56.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:55 smithi186 ceph-mon[33438]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:56.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:55 smithi186 ceph-mon[33438]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T04:52:56.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:55 smithi012 ceph-mon[29863]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:56.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:55 smithi012 ceph-mon[29863]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T04:52:56.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:55 smithi012 ceph-mon[41141]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:56.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:55 smithi012 ceph-mon[41141]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T04:52:56.944 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:52:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:57.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi012-78567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:52:57.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:56 smithi186 ceph-mon[33438]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T04:52:57.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:57.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:57.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi012-78567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:52:57.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[29863]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T04:52:57.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:57.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:57.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi012-78567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:52:57.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[41141]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T04:52:57.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:57.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:57 smithi186 ceph-mon[33438]: pgmap v503: 236 pgs: 5 creating+peering, 3 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:57 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:52:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:57 smithi186 ceph-mon[33438]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T04:52:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:58.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:58.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[29863]: pgmap v503: 236 pgs: 5 creating+peering, 3 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:58.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[29863]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[41141]: pgmap v503: 236 pgs: 5 creating+peering, 3 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[41141]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:58.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP::163:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:5d165639:::164:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f43765fc:::165:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b4c720e9:::166:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e694b040:::167:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:afa38db2:::168:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:77ba9f53:::169:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:87495034:::170:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7c96bf0e:::171:head 2024-03-26T04:52:58.951 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:dbe346cc:::172:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:e943ec24:::173:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f97a9c0c:::174:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:6f26e74d:::175:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4f95e106:::176:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:0e6f2f8f:::177:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:05db05f1:::178:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:38a78d66:::179:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:d095610b:::180:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:a1a9d709:::181:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:1e5d39db:::182:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:f7df4fb9:::183:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:03a7f161:::184:head 2024-03-26T04:52:58.952 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ba70721e:::185:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:28e5662d:::186:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:973d52de:::187:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:4303eb1c:::188:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:b990b48e:::189:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:29b8165b:::190:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:3547f197:::191:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:7e260936:::192:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:1abec7b1:::193:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:10fdda93:::194:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:15817eea:::195:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:770bab57:::196:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:ed9e13e7:::197:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:71471a8f:::198:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: checking for 278:10fb1d02:::199:head 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8033 ms) 2024-03-26T04:52:58.953 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: first is 1711428733 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,1711428738,1711428739,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,1711428738,1711428739,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,1711428738,1711428739,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,1711428738,1711428739,1711428741,1711428742,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,1711428738,1711428739,1711428741,1711428742,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428733,1711428735,1711428736,1711428738,1711428739,1711428741,1711428742,0 2024-03-26T04:52:58.954 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711428736,1711428738,1711428739,1711428741,1711428742,1711428744,1711428745,0 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: first now 1711428736, trimmed 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21363 ms) 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: foo0 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14073 ms) 2024-03-26T04:52:58.955 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-26T04:52:59.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:52:59.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]': finished 2024-03-26T04:52:59.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T04:52:59.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]': finished 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T04:52:59.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T04:52:59.653 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:52:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:59.709 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:52:59.709 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]': finished 2024-03-26T04:52:59.709 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-49"}]: dispatch 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T04:52:59.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:52:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]: dispatch 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ 2 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3034 ms) 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7023 ms) 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7042 ms) 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-26T04:52:59.963 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7028 ms) 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7025 ms) 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7046 ms) 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7031 ms) 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139407 ms total) 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (302591 ms total) 2024-03-26T04:52:59.964 INFO:tasks.workunit.client.0.smithi012.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-26T04:53:00.651 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[29863]: pgmap v506: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[29863]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[41141]: pgmap v506: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[41141]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:00.652 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:00.709 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:00 smithi186 ceph-mon[33438]: pgmap v506: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:00.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/539550162' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi012-78567-66"}]': finished 2024-03-26T04:53:00.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:00 smithi186 ceph-mon[33438]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T04:53:00.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:00.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:02.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:01 smithi186 ceph-mon[33438]: pgmap v509: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:02.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:02.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:01 smithi186 ceph-mon[33438]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T04:53:02.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:53:02.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:53:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[29863]: pgmap v509: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[29863]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T04:53:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:53:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:53:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[41141]: pgmap v509: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[41141]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T04:53:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:53:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:53:02.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:53:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:53:03.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:53:03.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[41141]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T04:53:03.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:03.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:53:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[29863]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T04:53:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:03.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:03.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:53:03.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:02 smithi186 ceph-mon[33438]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T04:53:03.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:03.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:04.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: pgmap v512: 228 pgs: 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:53:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]': finished 2024-03-26T04:53:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]: dispatch 2024-03-26T04:53:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T04:53:04.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]: dispatch 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: pgmap v512: 228 pgs: 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]': finished 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]: dispatch 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]: dispatch 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:04.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: pgmap v512: 228 pgs: 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-51"}]': finished 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]: dispatch 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]: dispatch 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:04.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:05.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:53:05.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]': finished 2024-03-26T04:53:05.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[29863]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T04:53:05.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:53:05.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:53:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:53:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]': finished 2024-03-26T04:53:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[41141]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T04:53:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:53:05.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:53:05.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:04 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:53:05.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-51", "mode": "writeback"}]': finished 2024-03-26T04:53:05.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:04 smithi186 ceph-mon[33438]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T04:53:05.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:53:05.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:53:06.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[29863]: pgmap v515: 228 pgs: 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:53:06.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:53:06.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:53:06.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[29863]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T04:53:06.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:53:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[41141]: pgmap v515: 228 pgs: 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:53:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:53:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:53:06.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[41141]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T04:53:06.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:53:06.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:05 smithi186 ceph-mon[33438]: pgmap v515: 228 pgs: 228 active+clean; 8.4 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:53:06.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:53:06.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:53:06.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:05 smithi186 ceph-mon[33438]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T04:53:06.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:53:06.986 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:53:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:07.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:53:07.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[29863]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T04:53:07.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:53:07.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:53:07.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:53:07.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[41141]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T04:53:07.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:53:07.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:53:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:53:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:06 smithi186 ceph-mon[33438]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T04:53:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:53:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:53:08.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[29863]: pgmap v518: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:08.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:53:08.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:53:08.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[29863]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[41141]: pgmap v518: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[41141]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:53:08.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:53:08.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:07 smithi186 ceph-mon[33438]: pgmap v518: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:08.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:07 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:53:08.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:53:08.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:07 smithi186 ceph-mon[33438]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T04:53:08.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:53:08.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:53:09.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:53:09.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:53:09.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[29863]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T04:53:09.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:53:09.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:53:09.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:53:09.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[41141]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T04:53:09.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:53:09.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:53:09.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:53:09.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:09 smithi186 ceph-mon[33438]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T04:53:09.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:53:10.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:10 smithi012 ceph-mon[29863]: pgmap v521: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:10.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:10 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:53:10.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:10 smithi012 ceph-mon[29863]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T04:53:10.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:10 smithi012 ceph-mon[41141]: pgmap v521: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:10.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:10 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:53:10.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:10 smithi012 ceph-mon[41141]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T04:53:10.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:10 smithi186 ceph-mon[33438]: pgmap v521: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:10.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:10 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:53:10.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:10 smithi186 ceph-mon[33438]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T04:53:11.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:11 smithi012 ceph-mon[29863]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:11.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:11 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:53:11.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:11 smithi012 ceph-mon[41141]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:11.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:11 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:53:11.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:11 smithi186 ceph-mon[33438]: pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:53:11.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:11 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:53:12.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:53:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:53:14.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:14.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:13 smithi186 ceph-mon[33438]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:14.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:14.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:13 smithi012 ceph-mon[29863]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:14.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:14.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:13 smithi012 ceph-mon[41141]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:16.203 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:15 smithi012 ceph-mon[29863]: pgmap v525: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:53:16.203 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:15 smithi012 ceph-mon[41141]: pgmap v525: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:53:16.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:15 smithi186 ceph-mon[33438]: pgmap v525: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:53:17.151 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:53:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:53:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:18.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:17 smithi186 ceph-mon[33438]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:18.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:17 smithi012 ceph-mon[29863]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:18.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:17 smithi012 ceph-mon[41141]: pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:19.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:19.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:19.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[29863]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:19.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:19.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:19.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:19.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:19.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[41141]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:19.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:19.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:19.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:19.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:19.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:19 smithi186 ceph-mon[33438]: pgmap v527: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:19.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:19.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:20.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:53:20.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[29863]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T04:53:20.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:20.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:20.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:53:20.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[41141]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T04:53:20.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:20.903 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:20.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:53:20.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:20 smithi186 ceph-mon[33438]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T04:53:20.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:20.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]': finished 2024-03-26T04:53:21.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T04:53:21.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[41141]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]': finished 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.903 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:21 smithi012 ceph-mon[29863]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:53:21.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]': finished 2024-03-26T04:53:21.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T04:53:21.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-51"}]: dispatch 2024-03-26T04:53:21.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:21 smithi186 ceph-mon[33438]: pgmap v530: 228 pgs: 228 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:53:22.611 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:53:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:53:22.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:22 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:53:22.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:22 smithi012 ceph-mon[29863]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T04:53:22.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:22 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:53:22.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:22 smithi012 ceph-mon[41141]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T04:53:22.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:22 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:53:22.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:22 smithi186 ceph-mon[33438]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T04:53:23.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:23.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[29863]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T04:53:23.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:23.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:23.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[29863]: pgmap v533: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:53:23.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:23.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[41141]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T04:53:23.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:23.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:23.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:23 smithi012 ceph-mon[41141]: pgmap v533: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:53:23.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:23.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:23 smithi186 ceph-mon[33438]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T04:53:23.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:23.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:23.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:23 smithi186 ceph-mon[33438]: pgmap v533: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:53:24.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:24.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:24.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T04:53:24.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:24.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-53"}]: dispatch 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-53"}]: dispatch 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-53"}]: dispatch 2024-03-26T04:53:24.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-53"}]: dispatch 2024-03-26T04:53:24.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:24.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:24.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T04:53:24.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:24.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:24.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-53"}]: dispatch 2024-03-26T04:53:24.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-53"}]: dispatch 2024-03-26T04:53:25.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:25 smithi012 ceph-mon[29863]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T04:53:25.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:25 smithi012 ceph-mon[29863]: pgmap v536: 196 pgs: 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:53:25.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:25 smithi012 ceph-mon[41141]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T04:53:25.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:25 smithi012 ceph-mon[41141]: pgmap v536: 196 pgs: 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:53:25.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:25 smithi186 ceph-mon[33438]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T04:53:25.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:25 smithi186 ceph-mon[33438]: pgmap v536: 196 pgs: 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:53:26.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-mon[29863]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T04:53:26.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:26.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:26.901 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:53:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:26.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-mon[41141]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T04:53:26.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:26.902 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:26.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:26 smithi186 ceph-mon[33438]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T04:53:26.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:26.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-55"}]: dispatch 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-55"}]: dispatch 2024-03-26T04:53:27.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[41141]: pgmap v539: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-55"}]: dispatch 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-55"}]: dispatch 2024-03-26T04:53:27.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:27 smithi012 ceph-mon[29863]: pgmap v539: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:53:27.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:27.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T04:53:27.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:27.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:27.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:27.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-55"}]: dispatch 2024-03-26T04:53:27.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-55"}]: dispatch 2024-03-26T04:53:27.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:27 smithi186 ceph-mon[33438]: pgmap v539: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:53:28.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:28 smithi186 ceph-mon[33438]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T04:53:29.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:28 smithi012 ceph-mon[29863]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T04:53:29.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:28 smithi012 ceph-mon[41141]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T04:53:29.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:29 smithi186 ceph-mon[33438]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T04:53:29.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:29.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:29.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:29 smithi186 ceph-mon[33438]: pgmap v542: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[29863]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[29863]: pgmap v542: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[41141]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:30.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:29 smithi012 ceph-mon[41141]: pgmap v542: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:53:30.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:30.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:30.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T04:53:30.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:30.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:30.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-57"}]: dispatch 2024-03-26T04:53:30.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-57"}]: dispatch 2024-03-26T04:53:31.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:31.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:31.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T04:53:31.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-57"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-57"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-57"}]: dispatch 2024-03-26T04:53:31.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-57"}]: dispatch 2024-03-26T04:53:31.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:31 smithi186 ceph-mon[33438]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T04:53:31.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:31 smithi186 ceph-mon[33438]: pgmap v545: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:32.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:31 smithi012 ceph-mon[29863]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T04:53:32.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:31 smithi012 ceph-mon[29863]: pgmap v545: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:32.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:31 smithi012 ceph-mon[41141]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T04:53:32.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:31 smithi012 ceph-mon[41141]: pgmap v545: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:32.697 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:53:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:53:32.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:32 smithi186 ceph-mon[33438]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T04:53:32.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:32.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:32 smithi012 ceph-mon[29863]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T04:53:33.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:32 smithi012 ceph-mon[41141]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T04:53:33.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-59"}]: dispatch 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-59"}]: dispatch 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: pgmap v548: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:33.960 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T04:53:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-59"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-59"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: pgmap v548: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:34.152 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:34.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-59"}]: dispatch 2024-03-26T04:53:34.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-59"}]: dispatch 2024-03-26T04:53:34.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: pgmap v548: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:53:34.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:34.153 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:34.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:34 smithi186 ceph-mon[33438]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T04:53:35.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:34 smithi012 ceph-mon[29863]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T04:53:35.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:34 smithi012 ceph-mon[41141]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T04:53:36.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[29863]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T04:53:36.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:36.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:36.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[29863]: pgmap v551: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:53:36.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[41141]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T04:53:36.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:36.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:36.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:35 smithi012 ceph-mon[41141]: pgmap v551: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:53:36.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:35 smithi186 ceph-mon[33438]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T04:53:36.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:36.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:36.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:35 smithi186 ceph-mon[33438]: pgmap v551: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:53:37.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:37.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:37.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[29863]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T04:53:37.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:37.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:37.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:53:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[41141]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:37.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:37.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:36 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:37.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:37.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:36 smithi186 ceph-mon[33438]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T04:53:37.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:37.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:37.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-61"}]: dispatch 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-61"}]: dispatch 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[29863]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:53:38.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:53:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T04:53:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-61"}]: dispatch 2024-03-26T04:53:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-61"}]: dispatch 2024-03-26T04:53:38.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:37 smithi012 ceph-mon[41141]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-61"}]: dispatch 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-61"}]: dispatch 2024-03-26T04:53:38.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:37 smithi186 ceph-mon[33438]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:53:39.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:38 smithi012 ceph-mon[29863]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T04:53:39.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:38 smithi012 ceph-mon[41141]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T04:53:39.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:38 smithi186 ceph-mon[33438]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[29863]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[29863]: pgmap v557: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[41141]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:40.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:40.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:39 smithi012 ceph-mon[41141]: pgmap v557: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:53:40.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:39 smithi186 ceph-mon[33438]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T04:53:40.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:40.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:53:40.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:39 smithi186 ceph-mon[33438]: pgmap v557: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[29863]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[41141]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T04:53:41.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:41.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:41.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:41.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:53:41.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:40 smithi186 ceph-mon[33438]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T04:53:41.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:53:41.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:41.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:53:42.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:53:42.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:41 smithi012 ceph-mon[29863]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T04:53:42.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:41 smithi012 ceph-mon[29863]: pgmap v560: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:42.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:53:42.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:41 smithi012 ceph-mon[41141]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T04:53:42.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:41 smithi012 ceph-mon[41141]: pgmap v560: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:42.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:53:42.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:41 smithi186 ceph-mon[33438]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T04:53:42.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:41 smithi186 ceph-mon[33438]: pgmap v560: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:53:42.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:53:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:53:43.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:42 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:43.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:42 smithi012 ceph-mon[29863]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T04:53:43.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:42 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:43.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:42 smithi012 ceph-mon[41141]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T04:53:43.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:42 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:53:43.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:42 smithi186 ceph-mon[33438]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T04:53:44.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:44.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:43 smithi012 ceph-mon[29863]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T04:53:44.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:43 smithi012 ceph-mon[29863]: pgmap v563: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:53:44.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:44.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:43 smithi012 ceph-mon[41141]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T04:53:44.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:43 smithi012 ceph-mon[41141]: pgmap v563: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:53:44.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:44.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:43 smithi186 ceph-mon[33438]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T04:53:44.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:43 smithi186 ceph-mon[33438]: pgmap v563: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:53:45.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:44 smithi012 ceph-mon[29863]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T04:53:45.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:44 smithi012 ceph-mon[41141]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T04:53:45.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:44 smithi186 ceph-mon[33438]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T04:53:46.052 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:45 smithi012 ceph-mon[29863]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T04:53:46.052 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:45 smithi012 ceph-mon[29863]: pgmap v566: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:53:46.052 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:45 smithi012 ceph-mon[41141]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T04:53:46.052 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:45 smithi012 ceph-mon[41141]: pgmap v566: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:53:46.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:45 smithi186 ceph-mon[33438]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T04:53:46.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:45 smithi186 ceph-mon[33438]: pgmap v566: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:53:47.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:53:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:53:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:48.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:47 smithi186 ceph-mon[33438]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:53:48.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:47 smithi186 ceph-mon[33438]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T04:53:48.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:47 smithi012 ceph-mon[29863]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:53:48.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:47 smithi012 ceph-mon[29863]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T04:53:48.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:47 smithi012 ceph-mon[41141]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:53:48.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:47 smithi012 ceph-mon[41141]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T04:53:49.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:53:49.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:49.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:53:49.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:49.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:53:49.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:50.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:49 smithi186 ceph-mon[33438]: pgmap v569: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:53:50.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:53:50.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:53:50.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[29863]: pgmap v569: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[41141]: pgmap v569: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:53:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:53:52.291 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:51 smithi186 ceph-mon[33438]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 861 B/s wr, 3 op/s 2024-03-26T04:53:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:51 smithi012 ceph-mon[29863]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 861 B/s wr, 3 op/s 2024-03-26T04:53:52.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:51 smithi012 ceph-mon[41141]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 861 B/s wr, 3 op/s 2024-03-26T04:53:52.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:53:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:53:54.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:54.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:53 smithi012 ceph-mon[29863]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 756 B/s wr, 3 op/s 2024-03-26T04:53:54.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:54.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:53 smithi012 ceph-mon[41141]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 756 B/s wr, 3 op/s 2024-03-26T04:53:54.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:54.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:53 smithi186 ceph-mon[33438]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 756 B/s wr, 3 op/s 2024-03-26T04:53:56.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:55 smithi012 ceph-mon[29863]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 614 B/s wr, 2 op/s 2024-03-26T04:53:56.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:55 smithi012 ceph-mon[29863]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T04:53:56.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:55 smithi012 ceph-mon[41141]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 614 B/s wr, 2 op/s 2024-03-26T04:53:56.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:55 smithi012 ceph-mon[41141]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T04:53:56.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:55 smithi186 ceph-mon[33438]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 614 B/s wr, 2 op/s 2024-03-26T04:53:56.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:55 smithi186 ceph-mon[33438]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T04:53:57.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:53:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:53:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:58.261 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:57 smithi186 ceph-mon[33438]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:58.261 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:57 smithi186 ceph-mon[33438]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T04:53:58.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:57 smithi012 ceph-mon[29863]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:58.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:57 smithi012 ceph-mon[29863]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T04:53:58.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:57 smithi012 ceph-mon[41141]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:53:58.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:57 smithi012 ceph-mon[41141]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T04:54:00.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:53:59 smithi012 ceph-mon[29863]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:00.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:53:59 smithi012 ceph-mon[41141]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:00.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:53:59 smithi186 ceph-mon[33438]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]: dispatch 2024-03-26T04:54:01.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]: dispatch 2024-03-26T04:54:01.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]: dispatch 2024-03-26T04:54:01.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]: dispatch 2024-03-26T04:54:01.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]: dispatch 2024-03-26T04:54:01.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]: dispatch 2024-03-26T04:54:01.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]: dispatch 2024-03-26T04:54:01.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]: dispatch 2024-03-26T04:54:01.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]: dispatch 2024-03-26T04:54:01.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]: dispatch 2024-03-26T04:54:01.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]: dispatch 2024-03-26T04:54:01.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]: dispatch 2024-03-26T04:54:01.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]: dispatch 2024-03-26T04:54:01.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]: dispatch 2024-03-26T04:54:01.460 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]: dispatch 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]': finished 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]': finished 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]': finished 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]': finished 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]': finished 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]': finished 2024-03-26T04:54:02.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-mon[33438]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]': finished 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]': finished 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]': finished 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]': finished 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]': finished 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]': finished 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[29863]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T04:54:02.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 4]}]': finished 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.15", "id": [7, 5]}]': finished 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.17", "id": [7, 4]}]': finished 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 0]}]': finished 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.14", "id": [5, 1]}]': finished 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "117.1c", "id": [5, 4]}]': finished 2024-03-26T04:54:02.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:02 smithi012 ceph-mon[41141]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T04:54:02.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:54:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:54:03.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:03 smithi012 ceph-mon[29863]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T04:54:03.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:03 smithi012 ceph-mon[41141]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T04:54:03.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:03 smithi186 ceph-mon[33438]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T04:54:04.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:04 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:04.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:04 smithi012 ceph-mon[29863]: pgmap v580: 228 pgs: 6 peering, 222 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:54:04.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:04.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:04 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:04.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:04 smithi012 ceph-mon[41141]: pgmap v580: 228 pgs: 6 peering, 222 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:54:04.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:04.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:04 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:04.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:04 smithi186 ceph-mon[33438]: pgmap v580: 228 pgs: 6 peering, 222 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:54:04.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:06.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:06 smithi012 ceph-mon[29863]: pgmap v581: 228 pgs: 6 peering, 222 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-03-26T04:54:06.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:06 smithi012 ceph-mon[29863]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T04:54:06.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:06 smithi012 ceph-mon[41141]: pgmap v581: 228 pgs: 6 peering, 222 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-03-26T04:54:06.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:06 smithi012 ceph-mon[41141]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T04:54:06.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:06 smithi186 ceph-mon[33438]: pgmap v581: 228 pgs: 6 peering, 222 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 698 B/s rd, 0 op/s 2024-03-26T04:54:06.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:06 smithi186 ceph-mon[33438]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T04:54:07.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:54:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:54:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:54:08.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:08 smithi012 ceph-mon[29863]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 16 B/s, 6 objects/s recovering 2024-03-26T04:54:08.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:08 smithi012 ceph-mon[29863]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T04:54:08.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:08 smithi012 ceph-mon[41141]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 16 B/s, 6 objects/s recovering 2024-03-26T04:54:08.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:08 smithi012 ceph-mon[41141]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T04:54:08.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:08 smithi186 ceph-mon[33438]: pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 16 B/s, 6 objects/s recovering 2024-03-26T04:54:08.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:08 smithi186 ceph-mon[33438]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T04:54:10.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:10 smithi012 ceph-mon[29863]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s; 14 B/s, 5 objects/s recovering 2024-03-26T04:54:10.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:10 smithi012 ceph-mon[41141]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s; 14 B/s, 5 objects/s recovering 2024-03-26T04:54:10.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:10 smithi186 ceph-mon[33438]: pgmap v585: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s; 14 B/s, 5 objects/s recovering 2024-03-26T04:54:11.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:11 smithi012 ceph-mon[41141]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 12 B/s, 4 objects/s recovering 2024-03-26T04:54:11.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:11 smithi012 ceph-mon[29863]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 12 B/s, 4 objects/s recovering 2024-03-26T04:54:11.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:11 smithi186 ceph-mon[33438]: pgmap v586: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 12 B/s, 4 objects/s recovering 2024-03-26T04:54:12.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:54:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:54:14.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:14.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:13 smithi012 ceph-mon[29863]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 12 B/s, 4 objects/s recovering 2024-03-26T04:54:14.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:14.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:13 smithi012 ceph-mon[41141]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 12 B/s, 4 objects/s recovering 2024-03-26T04:54:14.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:14.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:13 smithi186 ceph-mon[33438]: pgmap v587: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 12 B/s, 4 objects/s recovering 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[29863]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 9 B/s, 4 objects/s recovering 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-63"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-63"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[41141]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 9 B/s, 4 objects/s recovering 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-63"}]: dispatch 2024-03-26T04:54:16.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-63"}]: dispatch 2024-03-26T04:54:16.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:15 smithi186 ceph-mon[33438]: pgmap v588: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 9 B/s, 4 objects/s recovering 2024-03-26T04:54:16.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:16.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:16.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-63"}]: dispatch 2024-03-26T04:54:16.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-63"}]: dispatch 2024-03-26T04:54:16.980 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:54:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:54:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:54:17.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:16 smithi012 ceph-mon[29863]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T04:54:17.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:16 smithi012 ceph-mon[41141]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T04:54:17.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:16 smithi186 ceph-mon[33438]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[29863]: pgmap v590: 196 pgs: 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[29863]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[41141]: pgmap v590: 196 pgs: 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[41141]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:18.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:18.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:17 smithi186 ceph-mon[33438]: pgmap v590: 196 pgs: 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:18.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:17 smithi186 ceph-mon[33438]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T04:54:18.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:18.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[29863]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[41141]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T04:54:19.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:19.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:19.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:19.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:19.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:19.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:19 smithi186 ceph-mon[33438]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T04:54:19.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:19.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:19.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:19 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:19.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:20 smithi012 ceph-mon[29863]: pgmap v593: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:20.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:54:20.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:20 smithi012 ceph-mon[29863]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:20 smithi012 ceph-mon[41141]: pgmap v593: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:54:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:20 smithi012 ceph-mon[41141]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T04:54:20.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:20 smithi186 ceph-mon[33438]: pgmap v593: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:20.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:54:20.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:20 smithi186 ceph-mon[33438]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T04:54:21.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:21 smithi012 ceph-mon[29863]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T04:54:21.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:21 smithi012 ceph-mon[41141]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T04:54:21.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:21 smithi186 ceph-mon[33438]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T04:54:22.314 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:22 smithi186 ceph-mon[33438]: pgmap v596: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:22.314 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:22 smithi186 ceph-mon[33438]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T04:54:22.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:22 smithi012 ceph-mon[29863]: pgmap v596: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:22.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:22 smithi012 ceph-mon[29863]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T04:54:22.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:22 smithi012 ceph-mon[41141]: pgmap v596: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:22.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:22 smithi012 ceph-mon[41141]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T04:54:22.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:54:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:54:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:23 smithi012 ceph-mon[29863]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T04:54:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:23 smithi012 ceph-mon[41141]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T04:54:23.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:23 smithi186 ceph-mon[33438]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T04:54:24.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:24 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:24.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:24 smithi012 ceph-mon[29863]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-26T04:54:24.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:24 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:24.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:24 smithi012 ceph-mon[41141]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-26T04:54:24.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:24 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:24.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:24 smithi186 ceph-mon[33438]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-26T04:54:25.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:25 smithi012 ceph-mon[29863]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:54:25.416 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:25 smithi012 ceph-mon[41141]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:54:25.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:25 smithi186 ceph-mon[33438]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:54:27.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:54:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:54:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:54:28.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:27 smithi012 ceph-mon[29863]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-26T04:54:28.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:27 smithi012 ceph-mon[29863]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T04:54:28.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:27 smithi012 ceph-mon[41141]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-26T04:54:28.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:27 smithi012 ceph-mon[41141]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T04:54:28.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:27 smithi186 ceph-mon[33438]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-26T04:54:28.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:27 smithi186 ceph-mon[33438]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T04:54:30.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:29 smithi012 ceph-mon[29863]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:54:30.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:29 smithi012 ceph-mon[41141]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:54:30.462 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:29 smithi186 ceph-mon[33438]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:54:32.325 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:31 smithi186 ceph-mon[33438]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:32.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:31 smithi012 ceph-mon[29863]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:32.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:31 smithi012 ceph-mon[41141]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:32.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:54:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:54:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-65"}]: dispatch 2024-03-26T04:54:33.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-65"}]: dispatch 2024-03-26T04:54:33.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:33.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:33.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-65"}]: dispatch 2024-03-26T04:54:33.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-65"}]: dispatch 2024-03-26T04:54:33.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:33.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:33.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-65"}]: dispatch 2024-03-26T04:54:33.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-65"}]: dispatch 2024-03-26T04:54:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[29863]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:34.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[29863]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T04:54:34.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[41141]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[41141]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T04:54:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:33 smithi186 ceph-mon[33438]: pgmap v605: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:54:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:33 smithi186 ceph-mon[33438]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T04:54:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:35 smithi012 ceph-mon[29863]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T04:54:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:35 smithi012 ceph-mon[41141]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T04:54:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:35.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:35 smithi186 ceph-mon[33438]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T04:54:35.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:35.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:36.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[29863]: pgmap v608: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[29863]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[41141]: pgmap v608: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[41141]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T04:54:36.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:36.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:36 smithi186 ceph-mon[33438]: pgmap v608: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:36.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:36 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:54:36.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:36.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:36 smithi186 ceph-mon[33438]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T04:54:36.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:37.021 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:54:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:54:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:54:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[29863]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T04:54:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-67"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-67"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[41141]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-67"}]: dispatch 2024-03-26T04:54:37.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-67"}]: dispatch 2024-03-26T04:54:37.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:37 smithi186 ceph-mon[33438]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T04:54:37.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:37.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:37.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-67"}]: dispatch 2024-03-26T04:54:37.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-67"}]: dispatch 2024-03-26T04:54:38.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:38 smithi012 ceph-mon[29863]: pgmap v611: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:38.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:38 smithi012 ceph-mon[29863]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T04:54:38.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:38 smithi012 ceph-mon[41141]: pgmap v611: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:38.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:38 smithi012 ceph-mon[41141]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T04:54:38.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:38 smithi186 ceph-mon[33438]: pgmap v611: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:38.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:38 smithi186 ceph-mon[33438]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T04:54:39.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[29863]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[29863]: pgmap v614: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[41141]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:39.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:39 smithi012 ceph-mon[41141]: pgmap v614: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:54:39.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:39 smithi186 ceph-mon[33438]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T04:54:39.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:39.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:39.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:39 smithi186 ceph-mon[33438]: pgmap v614: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:54:40.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-69"}]: dispatch 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-69"}]: dispatch 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T04:54:40.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:40.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:40.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:40.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-69"}]: dispatch 2024-03-26T04:54:40.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-69"}]: dispatch 2024-03-26T04:54:40.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:40.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T04:54:40.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:40.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:40.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:54:40.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-69"}]: dispatch 2024-03-26T04:54:40.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-69"}]: dispatch 2024-03-26T04:54:41.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:41 smithi012 ceph-mon[29863]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T04:54:41.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:41 smithi012 ceph-mon[29863]: pgmap v617: 196 pgs: 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:41.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:41 smithi012 ceph-mon[41141]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T04:54:41.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:41 smithi012 ceph-mon[41141]: pgmap v617: 196 pgs: 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:41.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:41 smithi186 ceph-mon[33438]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T04:54:41.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:41 smithi186 ceph-mon[33438]: pgmap v617: 196 pgs: 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:54:42.338 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:42 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:54:42.338 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:42 smithi186 ceph-mon[33438]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T04:54:42.338 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:42.338 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:42.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:54:42.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[29863]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T04:54:42.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:42.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:42.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:54:42.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[41141]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T04:54:42.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:42.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:54:42.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:54:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:54:43.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[29863]: pgmap v620: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:43.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T04:54:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:43.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:43 smithi012 ceph-mon[41141]: pgmap v620: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:43.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:43 smithi186 ceph-mon[33438]: pgmap v620: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:54:44.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:54:44.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:44 smithi012 ceph-mon[29863]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T04:54:44.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:54:44.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:44 smithi012 ceph-mon[41141]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T04:54:44.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:54:44.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:44 smithi186 ceph-mon[33438]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T04:54:45.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:45 smithi012 ceph-mon[29863]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T04:54:45.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:45 smithi012 ceph-mon[29863]: pgmap v623: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:54:45.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:45 smithi012 ceph-mon[41141]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T04:54:45.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:45 smithi012 ceph-mon[41141]: pgmap v623: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:54:45.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:45 smithi186 ceph-mon[33438]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T04:54:45.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:45 smithi186 ceph-mon[33438]: pgmap v623: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:54:47.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:54:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:54:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:54:48.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:47 smithi012 ceph-mon[29863]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-26T04:54:48.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:47 smithi012 ceph-mon[41141]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-26T04:54:48.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:47 smithi186 ceph-mon[33438]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-26T04:54:49.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:49.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:49.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:54:50.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[29863]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-26T04:54:50.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:54:50.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[41141]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:54:50.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:54:50.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:49 smithi186 ceph-mon[33438]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-26T04:54:50.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:54:50.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:54:50.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:54:50.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:54:52.347 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:51 smithi186 ceph-mon[33438]: pgmap v626: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:54:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:51 smithi012 ceph-mon[29863]: pgmap v626: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:54:52.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:51 smithi012 ceph-mon[41141]: pgmap v626: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:54:52.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:54:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:54:53.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:53.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:53 smithi012 ceph-mon[29863]: pgmap v627: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T04:54:53.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:53.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:53 smithi012 ceph-mon[41141]: pgmap v627: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T04:54:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:54:53.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:53 smithi186 ceph-mon[33438]: pgmap v627: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T04:54:56.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:55 smithi012 ceph-mon[29863]: pgmap v628: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T04:54:56.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:55 smithi012 ceph-mon[41141]: pgmap v628: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T04:54:56.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:55 smithi186 ceph-mon[33438]: pgmap v628: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T04:54:57.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:54:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:54:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:54:58.261 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:57 smithi186 ceph-mon[33438]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T04:54:58.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:57 smithi012 ceph-mon[29863]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T04:54:58.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:57 smithi012 ceph-mon[41141]: pgmap v629: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T04:55:00.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:54:59 smithi012 ceph-mon[29863]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:00.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:54:59 smithi012 ceph-mon[41141]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:00.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:54:59 smithi186 ceph-mon[33438]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:02.355 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:01 smithi186 ceph-mon[33438]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:02.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:01 smithi012 ceph-mon[29863]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:02.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:01 smithi012 ceph-mon[41141]: pgmap v631: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:02.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:55:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:55:04.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:04.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:03 smithi012 ceph-mon[29863]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:04.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:04.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:04.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:03 smithi012 ceph-mon[41141]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:04.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:04.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:04.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:03 smithi186 ceph-mon[33438]: pgmap v632: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:04.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:05.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:05.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:05.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-71"}]: dispatch 2024-03-26T04:55:05.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-71"}]: dispatch 2024-03-26T04:55:05.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:05.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:05.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-71"}]: dispatch 2024-03-26T04:55:05.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-71"}]: dispatch 2024-03-26T04:55:05.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:05.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:05.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-71"}]: dispatch 2024-03-26T04:55:05.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-71"}]: dispatch 2024-03-26T04:55:06.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:06 smithi012 ceph-mon[29863]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:06.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:06 smithi012 ceph-mon[29863]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T04:55:06.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:06 smithi012 ceph-mon[41141]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:06.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:06 smithi012 ceph-mon[41141]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T04:55:06.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:06 smithi186 ceph-mon[33438]: pgmap v633: 228 pgs: 228 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:55:06.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:06 smithi186 ceph-mon[33438]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T04:55:07.021 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:55:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:55:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:55:07.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[29863]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T04:55:07.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:07.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:07.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:07.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[29863]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T04:55:07.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[41141]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T04:55:07.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:07.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:07.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:07.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:07 smithi012 ceph-mon[41141]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T04:55:07.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:07 smithi186 ceph-mon[33438]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T04:55:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:07.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:07 smithi186 ceph-mon[33438]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T04:55:08.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[29863]: pgmap v636: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:08.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:08.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:08.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[29863]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T04:55:08.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[41141]: pgmap v636: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:08.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:08.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:08.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:08 smithi012 ceph-mon[41141]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T04:55:08.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:08 smithi186 ceph-mon[33438]: pgmap v636: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:08.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:08.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:08 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:08.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:08 smithi186 ceph-mon[33438]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T04:55:10.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:10 smithi012 ceph-mon[29863]: pgmap v639: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:10.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:10 smithi012 ceph-mon[29863]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T04:55:10.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:10 smithi012 ceph-mon[41141]: pgmap v639: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:10.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:10 smithi012 ceph-mon[41141]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T04:55:10.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:10 smithi186 ceph-mon[33438]: pgmap v639: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:10.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:10 smithi186 ceph-mon[33438]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T04:55:12.366 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:12 smithi186 ceph-mon[33438]: pgmap v641: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:55:12.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:12 smithi012 ceph-mon[29863]: pgmap v641: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:55:12.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:12 smithi012 ceph-mon[41141]: pgmap v641: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:55:12.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:55:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:55:13.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:13 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:13.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:13 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:13.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:13 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:14.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:14 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:14.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:14 smithi012 ceph-mon[29863]: pgmap v642: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:55:14.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:14 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:14.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:14 smithi012 ceph-mon[41141]: pgmap v642: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:55:14.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:14 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:14.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:14 smithi186 ceph-mon[33438]: pgmap v642: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:55:16.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:16 smithi012 ceph-mon[29863]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 903 B/s wr, 3 op/s 2024-03-26T04:55:16.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:16 smithi012 ceph-mon[41141]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 903 B/s wr, 3 op/s 2024-03-26T04:55:16.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:16 smithi186 ceph-mon[33438]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 903 B/s wr, 3 op/s 2024-03-26T04:55:17.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:55:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:55:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:55:18.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:18 smithi012 ceph-mon[29863]: pgmap v644: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 803 B/s wr, 3 op/s 2024-03-26T04:55:18.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:18 smithi012 ceph-mon[41141]: pgmap v644: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 803 B/s wr, 3 op/s 2024-03-26T04:55:18.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:18 smithi186 ceph-mon[33438]: pgmap v644: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 803 B/s wr, 3 op/s 2024-03-26T04:55:19.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:19.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:19.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[29863]: pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-26T04:55:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-73"}]: dispatch 2024-03-26T04:55:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-73"}]: dispatch 2024-03-26T04:55:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[41141]: pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-26T04:55:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-73"}]: dispatch 2024-03-26T04:55:20.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-73"}]: dispatch 2024-03-26T04:55:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:20 smithi186 ceph-mon[33438]: pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-26T04:55:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:20.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:20.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-73"}]: dispatch 2024-03-26T04:55:20.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-73"}]: dispatch 2024-03-26T04:55:21.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:21 smithi012 ceph-mon[41141]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T04:55:21.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:21 smithi012 ceph-mon[41141]: pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2024-03-26T04:55:21.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:21 smithi012 ceph-mon[29863]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T04:55:21.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:21 smithi012 ceph-mon[29863]: pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2024-03-26T04:55:21.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:21 smithi186 ceph-mon[33438]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T04:55:21.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:21 smithi186 ceph-mon[33438]: pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2024-03-26T04:55:22.372 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:22 smithi186 ceph-mon[33438]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T04:55:22.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:22.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:22 smithi012 ceph-mon[41141]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T04:55:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:22.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:22 smithi012 ceph-mon[29863]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T04:55:22.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:22.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:22.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:55:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:55:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[29863]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T04:55:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[29863]: pgmap v650: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:23.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:23.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[41141]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T04:55:23.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:23.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:23.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:23 smithi012 ceph-mon[41141]: pgmap v650: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:23 smithi186 ceph-mon[33438]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T04:55:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:23.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:23.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:23 smithi186 ceph-mon[33438]: pgmap v650: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:24.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:24 smithi012 ceph-mon[41141]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T04:55:24.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:24 smithi012 ceph-mon[29863]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T04:55:24.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:24 smithi186 ceph-mon[33438]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T04:55:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[41141]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T04:55:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-75"}]: dispatch 2024-03-26T04:55:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-75"}]: dispatch 2024-03-26T04:55:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[41141]: pgmap v653: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:55:25.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[29863]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T04:55:25.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:25.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:25.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-75"}]: dispatch 2024-03-26T04:55:25.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-75"}]: dispatch 2024-03-26T04:55:25.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:25 smithi012 ceph-mon[29863]: pgmap v653: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:55:25.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:25 smithi186 ceph-mon[33438]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T04:55:25.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:25.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:25.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-75"}]: dispatch 2024-03-26T04:55:25.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-75"}]: dispatch 2024-03-26T04:55:25.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:25 smithi186 ceph-mon[33438]: pgmap v653: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:55:26.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:26 smithi012 ceph-mon[29863]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T04:55:26.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:26 smithi012 ceph-mon[41141]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T04:55:26.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:26 smithi186 ceph-mon[33438]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T04:55:27.087 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:55:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:55:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:55:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[29863]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T04:55:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[29863]: pgmap v656: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:55:27.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[41141]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T04:55:27.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:27.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:27.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:27 smithi012 ceph-mon[41141]: pgmap v656: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:55:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:27 smithi186 ceph-mon[33438]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T04:55:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:27 smithi186 ceph-mon[33438]: pgmap v656: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:55:28.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:28 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:28.360 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:28.360 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:28 smithi186 ceph-mon[33438]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T04:55:28.360 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[29863]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:28.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[41141]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T04:55:28.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:29.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[29863]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T04:55:29.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:29.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:29.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-77"}]: dispatch 2024-03-26T04:55:29.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-77"}]: dispatch 2024-03-26T04:55:29.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[29863]: pgmap v659: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:55:29.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[41141]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T04:55:29.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:29.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:29.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-77"}]: dispatch 2024-03-26T04:55:29.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-77"}]: dispatch 2024-03-26T04:55:29.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:29 smithi012 ceph-mon[41141]: pgmap v659: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:55:29.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:29 smithi186 ceph-mon[33438]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T04:55:29.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:29.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:29.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-77"}]: dispatch 2024-03-26T04:55:29.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-77"}]: dispatch 2024-03-26T04:55:29.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:29 smithi186 ceph-mon[33438]: pgmap v659: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:55:30.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:30 smithi012 ceph-mon[29863]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T04:55:30.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:30 smithi012 ceph-mon[41141]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T04:55:30.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:30 smithi186 ceph-mon[33438]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T04:55:31.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[29863]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T04:55:31.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:31.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:31.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[29863]: pgmap v662: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[41141]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T04:55:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:31.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:31.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:31 smithi012 ceph-mon[41141]: pgmap v662: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:31.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:31 smithi186 ceph-mon[33438]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T04:55:31.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:31.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:31.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:31 smithi186 ceph-mon[33438]: pgmap v662: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:55:32.382 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:32.383 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:32 smithi186 ceph-mon[33438]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T04:55:32.383 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:32.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:32.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:32 smithi012 ceph-mon[29863]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T04:55:32.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:32.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:32.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:32 smithi012 ceph-mon[41141]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T04:55:32.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:32.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:55:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:55:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:33 smithi012 ceph-mon[29863]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T04:55:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:33 smithi012 ceph-mon[29863]: pgmap v665: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-26T04:55:33.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:33 smithi012 ceph-mon[41141]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T04:55:33.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:33.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:33 smithi012 ceph-mon[41141]: pgmap v665: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-26T04:55:33.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:33 smithi186 ceph-mon[33438]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T04:55:33.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:33.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:33 smithi186 ceph-mon[33438]: pgmap v665: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.10"}]: dispatch 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.10"}]: dispatch 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: 305.10 deep-scrub starts 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: 305.10 deep-scrub ok 2024-03-26T04:55:34.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.10"}]: dispatch 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.10"}]: dispatch 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: 305.10 deep-scrub starts 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: 305.10 deep-scrub ok 2024-03-26T04:55:34.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T04:55:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.10"}]: dispatch 2024-03-26T04:55:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.10"}]: dispatch 2024-03-26T04:55:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: 305.10 deep-scrub starts 2024-03-26T04:55:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: 305.10 deep-scrub ok 2024-03-26T04:55:34.459 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:35 smithi012 ceph-mon[29863]: pgmap v667: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 1.0 KiB/s wr, 151 op/s 2024-03-26T04:55:35.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:35 smithi012 ceph-mon[41141]: pgmap v667: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 1.0 KiB/s wr, 151 op/s 2024-03-26T04:55:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:35 smithi186 ceph-mon[33438]: pgmap v667: 228 pgs: 228 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 1.0 KiB/s wr, 151 op/s 2024-03-26T04:55:37.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:55:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:55:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:55:38.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:37 smithi012 ceph-mon[29863]: pgmap v668: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.2 KiB/s wr, 123 op/s 2024-03-26T04:55:38.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:37 smithi012 ceph-mon[41141]: pgmap v668: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.2 KiB/s wr, 123 op/s 2024-03-26T04:55:38.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:37 smithi186 ceph-mon[33438]: pgmap v668: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.2 KiB/s wr, 123 op/s 2024-03-26T04:55:40.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:39 smithi012 ceph-mon[29863]: pgmap v669: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 916 B/s wr, 94 op/s 2024-03-26T04:55:40.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:39 smithi012 ceph-mon[41141]: pgmap v669: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 916 B/s wr, 94 op/s 2024-03-26T04:55:40.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:39 smithi186 ceph-mon[33438]: pgmap v669: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 916 B/s wr, 94 op/s 2024-03-26T04:55:42.393 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:42 smithi186 ceph-mon[33438]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 232 B/s wr, 1 op/s 2024-03-26T04:55:42.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:42 smithi012 ceph-mon[29863]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 232 B/s wr, 1 op/s 2024-03-26T04:55:42.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:42 smithi012 ceph-mon[41141]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 232 B/s wr, 1 op/s 2024-03-26T04:55:42.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:55:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:55:44.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:44 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:44.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:44 smithi012 ceph-mon[29863]: pgmap v671: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:55:44.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:44 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:44.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:44 smithi012 ceph-mon[41141]: pgmap v671: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:55:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:44 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:44 smithi186 ceph-mon[33438]: pgmap v671: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:55:46.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:46 smithi012 ceph-mon[29863]: pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 173 B/s wr, 1 op/s 2024-03-26T04:55:46.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:46 smithi012 ceph-mon[41141]: pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 173 B/s wr, 1 op/s 2024-03-26T04:55:46.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:46 smithi186 ceph-mon[33438]: pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 173 B/s wr, 1 op/s 2024-03-26T04:55:47.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:55:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:55:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:55:48.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:48 smithi012 ceph-mon[29863]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T04:55:48.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:48 smithi012 ceph-mon[41141]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T04:55:48.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:48 smithi186 ceph-mon[33438]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-03-26T04:55:49.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:49.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:49.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:55:50.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:50 smithi012 ceph-mon[29863]: pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:55:50.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:50 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:50.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:50 smithi012 ceph-mon[41141]: pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:55:50.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:50 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:50.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:50 smithi186 ceph-mon[33438]: pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:55:50.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:50 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:55:51.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:55:51.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:55:51.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:55:51.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:55:51.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:55:51.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:55:51.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:55:51.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:55:51.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:55:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:52 smithi012 ceph-mon[29863]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:55:52.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:52 smithi012 ceph-mon[41141]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:55:52.404 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:52 smithi186 ceph-mon[33438]: pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:55:52.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:55:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:55:53.458 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T04:55:53.186+0000 7feddc8dd700 -1 snap_mapper.add_oid found existing snaps mapped on 305:0abebd0c:test-rados-api-smithi012-78824-80::foo:2, removing 2024-03-26T04:55:53.603 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T04:55:53.185+0000 7fbe1906e700 -1 snap_mapper.add_oid found existing snaps mapped on 305:0abebd0c:test-rados-api-smithi012-78824-80::foo:2, removing 2024-03-26T04:55:53.603 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2[57219]: 2024-03-26T04:55:53.185+0000 7fa9c4afc700 -1 snap_mapper.add_oid found existing snaps mapped on 305:0abebd0c:test-rados-api-smithi012-78824-80::foo:2, removing 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[41141]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-79"}]: dispatch 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-79"}]: dispatch 2024-03-26T04:55:53.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:53.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[29863]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:55:53.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:53.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:53.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-79"}]: dispatch 2024-03-26T04:55:53.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:53 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-79"}]: dispatch 2024-03-26T04:55:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:55:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-mon[33438]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:55:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:53.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-79"}]: dispatch 2024-03-26T04:55:53.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:53 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-79"}]: dispatch 2024-03-26T04:55:54.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:54 smithi012 ceph-mon[29863]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T04:55:54.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:54 smithi012 ceph-mon[41141]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T04:55:54.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:54 smithi186 ceph-mon[33438]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[29863]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[29863]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[41141]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:55.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:55 smithi012 ceph-mon[41141]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:55:55.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:55 smithi186 ceph-mon[33438]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T04:55:55.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:55.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:55 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:55:55.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:55 smithi186 ceph-mon[33438]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:55:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[29863]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:55:56.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:55:56.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:56.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:56.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[41141]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T04:55:56.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:56.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:55:56.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:55:56.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:56 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:55:56.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:55:56.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:56 smithi186 ceph-mon[33438]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T04:55:56.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:55:56.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:55:56.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:55:57.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:55:57.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:57.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[29863]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T04:55:57.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:57.901 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[29863]: pgmap v682: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T04:55:57.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:55:57.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:57.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[41141]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T04:55:57.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:57.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:57 smithi012 ceph-mon[41141]: pgmap v682: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T04:55:57.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:55:57.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:57.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:57 smithi186 ceph-mon[33438]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T04:55:57.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:55:57.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:57 smithi186 ceph-mon[33438]: pgmap v682: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[29863]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[41141]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T04:55:58.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:58 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:55:58.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:58 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:55:58.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:55:58.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:58 smithi186 ceph-mon[33438]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T04:55:58.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:58 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:55:59.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[29863]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:55:59.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:55:59 smithi012 ceph-mon[41141]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T04:55:59.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:55:59.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T04:55:59.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:55:59.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:55:59.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:55:59.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:55:59.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:55:59 smithi186 ceph-mon[33438]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T04:56:00.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:56:00.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:56:00.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:56:00.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:56:00.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:56:00.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:56:01.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T04:56:01.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T04:56:01.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:56:01.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:56:01.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:01.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]: dispatch 2024-03-26T04:56:01.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:01 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]: dispatch 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]: dispatch 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]: dispatch 2024-03-26T04:56:02.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T04:56:02.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T04:56:02.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:56:02.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:56:02.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:02.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]: dispatch 2024-03-26T04:56:02.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:01 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]: dispatch 2024-03-26T04:56:02.709 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:56:02.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T04:56:02.710 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]': finished 2024-03-26T04:56:02.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-mon[33438]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T04:56:02.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:02.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:02.711 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:02 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]': finished 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[29863]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:03.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T04:56:03.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.2", "id": [7, 2]}]': finished 2024-03-26T04:56:03.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[41141]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T04:56:03.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:03.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:03.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:02 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:03.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:03.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:03.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T04:56:03.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:03.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:03.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-81"}]: dispatch 2024-03-26T04:56:03.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-81"}]: dispatch 2024-03-26T04:56:03.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:03 smithi186 ceph-mon[33438]: pgmap v691: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T04:56:04.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:04.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:04.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T04:56:04.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-81"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-81"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[29863]: pgmap v691: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-81"}]: dispatch 2024-03-26T04:56:04.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-81"}]: dispatch 2024-03-26T04:56:04.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:03 smithi012 ceph-mon[41141]: pgmap v691: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-03-26T04:56:04.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:04 smithi186 ceph-mon[33438]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T04:56:04.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:05.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:04 smithi012 ceph-mon[29863]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T04:56:05.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:05.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:04 smithi012 ceph-mon[41141]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T04:56:05.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:05.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:05 smithi186 ceph-mon[33438]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T04:56:05.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:05.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:05.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:05 smithi186 ceph-mon[33438]: pgmap v694: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.0 KiB/s wr, 21 op/s 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[29863]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[29863]: pgmap v694: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.0 KiB/s wr, 21 op/s 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[41141]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:06.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:05 smithi012 ceph-mon[41141]: pgmap v694: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.0 KiB/s wr, 21 op/s 2024-03-26T04:56:06.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:06.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:06 smithi186 ceph-mon[33438]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T04:56:06.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:06.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:06.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:07.150 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:56:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:56:07.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:07.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[29863]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T04:56:07.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:07.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:07.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:07.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:07.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[41141]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T04:56:07.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:07.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:07.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:07.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:07.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:07 smithi186 ceph-mon[33438]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T04:56:07.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:07.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:07 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:07.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:07 smithi186 ceph-mon[33438]: pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-03-26T04:56:07.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:07 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[29863]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[29863]: pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[41141]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T04:56:08.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:08.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:08.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[41141]: pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-03-26T04:56:08.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:07 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:09.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:56:09.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[29863]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T04:56:09.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:09.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:09.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:56:09.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[41141]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T04:56:09.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:09.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:09.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:56:09.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:08 smithi186 ceph-mon[33438]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T04:56:09.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:09.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:10.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:10.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[29863]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T04:56:10.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:10.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:10.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[29863]: pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-03-26T04:56:10.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:10.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:10.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[41141]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T04:56:10.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:10.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:10.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[41141]: pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-03-26T04:56:10.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:10.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:10.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:09 smithi186 ceph-mon[33438]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T04:56:10.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:10.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:10.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:09 smithi186 ceph-mon[33438]: pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-03-26T04:56:10.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:11.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:10 smithi012 ceph-mon[29863]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T04:56:11.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:10 smithi012 ceph-mon[41141]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T04:56:11.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:10 smithi186 ceph-mon[33438]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T04:56:12.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:11 smithi012 ceph-mon[29863]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T04:56:12.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:11 smithi012 ceph-mon[29863]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:12.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:11 smithi012 ceph-mon[41141]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T04:56:12.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:11 smithi012 ceph-mon[41141]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:12.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:11 smithi186 ceph-mon[33438]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T04:56:12.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:11 smithi186 ceph-mon[33438]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:12.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[41141]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-83"}]: dispatch 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-83"}]: dispatch 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[29863]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:13.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:13.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-83"}]: dispatch 2024-03-26T04:56:13.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-83"}]: dispatch 2024-03-26T04:56:13.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:12 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:13.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-mon[33438]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T04:56:13.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:13.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:13.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-83"}]: dispatch 2024-03-26T04:56:13.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-83"}]: dispatch 2024-03-26T04:56:13.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:12 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:14.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:14.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:13 smithi012 ceph-mon[41141]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T04:56:14.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:13 smithi012 ceph-mon[41141]: pgmap v706: 196 pgs: 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:14.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:14.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:13 smithi012 ceph-mon[29863]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T04:56:14.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:13 smithi012 ceph-mon[29863]: pgmap v706: 196 pgs: 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:14.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:14.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:13 smithi186 ceph-mon[33438]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T04:56:14.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:13 smithi186 ceph-mon[33438]: pgmap v706: 196 pgs: 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:15.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:14 smithi012 ceph-mon[41141]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T04:56:15.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:15.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:14 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:15.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:14 smithi012 ceph-mon[29863]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T04:56:15.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:15.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:14 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:15.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:14 smithi186 ceph-mon[33438]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T04:56:15.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:15.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:14 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:16.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:16.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[29863]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T04:56:16.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:16.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:16.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:16.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[29863]: pgmap v709: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:16.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:16.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[41141]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T04:56:16.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:16.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:16.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:16.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:15 smithi012 ceph-mon[41141]: pgmap v709: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:16.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:16.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:15 smithi186 ceph-mon[33438]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T04:56:16.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:16.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:16.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:15 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:16.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:15 smithi186 ceph-mon[33438]: pgmap v709: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[29863]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T04:56:17.054 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:17.054 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:56:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:56:17.055 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:17.055 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:17.055 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[41141]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T04:56:17.055 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:16 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:17.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:17.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:17.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:16 smithi186 ceph-mon[33438]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T04:56:17.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:16 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:18.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:56:18.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:18.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T04:56:18.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:18.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: pgmap v712: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T04:56:18.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:18.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:18.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[29863]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T04:56:18.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: pgmap v712: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:18.152 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:17 smithi012 ceph-mon[41141]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_tier","val": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: pgmap v712: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:18.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:17 smithi186 ceph-mon[33438]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T04:56:19.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[29863]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:19.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:18 smithi012 ceph-mon[41141]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T04:56:19.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:19.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:18 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:19.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:18 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:19.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:18 smithi186 ceph-mon[33438]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T04:56:20.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:19 smithi012 ceph-mon[29863]: pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T04:56:20.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:19 smithi012 ceph-mon[41141]: pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T04:56:20.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:19 smithi186 ceph-mon[33438]: pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-26T04:56:21.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:20 smithi012 ceph-mon[29863]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T04:56:21.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:20 smithi012 ceph-mon[41141]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T04:56:21.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:20 smithi186 ceph-mon[33438]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[29863]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-85"}]: dispatch 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-85"}]: dispatch 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[29863]: pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[41141]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T04:56:22.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:22.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:22.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-85"}]: dispatch 2024-03-26T04:56:22.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-85"}]: dispatch 2024-03-26T04:56:22.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:21 smithi012 ceph-mon[41141]: pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:22.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:21 smithi186 ceph-mon[33438]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T04:56:22.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:22.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:22.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-85"}]: dispatch 2024-03-26T04:56:22.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:21 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-85"}]: dispatch 2024-03-26T04:56:22.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:21 smithi186 ceph-mon[33438]: pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:22.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:56:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:56:23.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:22 smithi012 ceph-mon[29863]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T04:56:23.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:22 smithi012 ceph-mon[41141]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T04:56:23.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:22 smithi186 ceph-mon[33438]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T04:56:23.822 INFO:tasks.workunit.client.0.smithi012.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (17835 ms) 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22670 ms) 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3016 ms) 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3019 ms) 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3020 ms) 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3034 ms) 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-26T04:56:23.823 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4022 ms) 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38253 ms) 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17008 ms) 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4035 ms) 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3023 ms) 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-26T04:56:23.824 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24948 ms) 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15041 ms) 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5042 ms) 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4026 ms) 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: done waiting 2024-03-26T04:56:23.825 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24502 ms) 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10072 ms) 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9060 ms) 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9058 ms) 2024-03-26T04:56:23.826 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-26T04:56:24.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[29863]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[29863]: pgmap v721: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[41141]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:24.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:23 smithi012 ceph-mon[41141]: pgmap v721: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:56:24.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:24.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:23 smithi186 ceph-mon[33438]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T04:56:24.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:24.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:24.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:23 smithi186 ceph-mon[33438]: pgmap v721: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:56:25.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:25.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:25.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[29863]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T04:56:25.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:25.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:25.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:25.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[41141]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T04:56:25.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:25.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:24 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:25.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:25.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:24 smithi186 ceph-mon[33438]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T04:56:25.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: pgmap v724: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]: dispatch 2024-03-26T04:56:26.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]: dispatch 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: pgmap v724: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]: dispatch 2024-03-26T04:56:26.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]: dispatch 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: pgmap v724: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]: dispatch 2024-03-26T04:56:26.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]: dispatch 2024-03-26T04:56:27.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:56:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:56:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]': finished 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[29863]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[29863]: pgmap v727: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]': finished 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[41141]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:28.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:28.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:27 smithi012 ceph-mon[41141]: pgmap v727: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:56:28.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_tier","val": "test-rados-api-smithi012-78824-89-test-flush"}]': finished 2024-03-26T04:56:28.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:27 smithi186 ceph-mon[33438]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T04:56:28.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:28.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:56:28.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:27 smithi186 ceph-mon[33438]: pgmap v727: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:56:29.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[29863]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[41141]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:29.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:29.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:56:29.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:28 smithi186 ceph-mon[33438]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T04:56:29.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:29.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:56:30.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:30.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:29 smithi012 ceph-mon[29863]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T04:56:30.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:29 smithi012 ceph-mon[29863]: pgmap v730: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:56:30.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:30.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:29 smithi012 ceph-mon[41141]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T04:56:30.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:29 smithi012 ceph-mon[41141]: pgmap v730: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:56:30.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:56:30.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:29 smithi186 ceph-mon[33438]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T04:56:30.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:29 smithi186 ceph-mon[33438]: pgmap v730: 260 pgs: 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:56:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[41141]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-87"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-87"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[29863]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-87"}]: dispatch 2024-03-26T04:56:31.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-87"}]: dispatch 2024-03-26T04:56:31.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:30 smithi186 ceph-mon[33438]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T04:56:31.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:31.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:31.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-87"}]: dispatch 2024-03-26T04:56:31.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-87"}]: dispatch 2024-03-26T04:56:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:31 smithi012 ceph-mon[29863]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T04:56:32.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:31 smithi012 ceph-mon[29863]: pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:32.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:31 smithi012 ceph-mon[41141]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T04:56:32.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:31 smithi012 ceph-mon[41141]: pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:32.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:31 smithi186 ceph-mon[33438]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T04:56:32.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:31 smithi186 ceph-mon[33438]: pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:32.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:56:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[29863]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[41141]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T04:56:33.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:33.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:33.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:32 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:33.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:32 smithi186 ceph-mon[33438]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T04:56:33.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:33.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:34.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:34.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:33 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:34.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:34.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:34.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T04:56:34.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:34.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:34.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:56:34.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:34.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:33 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:35.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:35.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:34 smithi012 ceph-mon[29863]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T04:56:35.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:35.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:34 smithi012 ceph-mon[41141]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T04:56:35.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:56:35.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:34 smithi186 ceph-mon[33438]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T04:56:36.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:35 smithi012 ceph-mon[29863]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T04:56:36.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:35 smithi012 ceph-mon[29863]: pgmap v739: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:36.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:35 smithi012 ceph-mon[41141]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T04:56:36.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:35 smithi012 ceph-mon[41141]: pgmap v739: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:36.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:35 smithi186 ceph-mon[33438]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T04:56:36.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:35 smithi186 ceph-mon[33438]: pgmap v739: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:56:37.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[29863]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T04:56:37.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-90"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-90"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[41141]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-90"}]: dispatch 2024-03-26T04:56:37.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-90"}]: dispatch 2024-03-26T04:56:37.151 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:56:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:56:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:56:37.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:36 smithi186 ceph-mon[33438]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T04:56:37.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:37.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:37.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-90"}]: dispatch 2024-03-26T04:56:37.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-90"}]: dispatch 2024-03-26T04:56:38.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:37 smithi012 ceph-mon[29863]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T04:56:38.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:37 smithi012 ceph-mon[29863]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:56:38.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:37 smithi012 ceph-mon[41141]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T04:56:38.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:37 smithi012 ceph-mon[41141]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:56:38.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:37 smithi186 ceph-mon[33438]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T04:56:38.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:37 smithi186 ceph-mon[33438]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:56:39.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:38 smithi186 ceph-mon[33438]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T04:56:39.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:39.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:39.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:38 smithi012 ceph-mon[41141]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T04:56:39.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:39.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:39.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:38 smithi012 ceph-mon[29863]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T04:56:39.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:39.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:40.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:40.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:39 smithi186 ceph-mon[33438]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T04:56:40.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:40.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:39 smithi186 ceph-mon[33438]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[29863]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[29863]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[41141]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:40.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:39 smithi012 ceph-mon[41141]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:56:41.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:40 smithi186 ceph-mon[33438]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T04:56:41.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:40 smithi012 ceph-mon[29863]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T04:56:41.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:40 smithi012 ceph-mon[41141]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T04:56:42.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-mon[33438]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T04:56:42.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-mon[33438]: pgmap v748: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:42.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:42.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:42.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-92"}]: dispatch 2024-03-26T04:56:42.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-92"}]: dispatch 2024-03-26T04:56:42.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[29863]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[29863]: pgmap v748: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-92"}]: dispatch 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-92"}]: dispatch 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[41141]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[41141]: pgmap v748: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:56:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-92"}]: dispatch 2024-03-26T04:56:42.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-92"}]: dispatch 2024-03-26T04:56:42.707 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:56:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:56:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:43 smithi186 ceph-mon[33438]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T04:56:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:43 smithi186 ceph-mon[33438]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T04:56:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:43 smithi186 ceph-mon[33438]: pgmap v751: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[29863]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[29863]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[29863]: pgmap v751: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:56:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[41141]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T04:56:43.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:43.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[41141]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T04:56:43.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:43.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:56:43.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:43 smithi012 ceph-mon[41141]: pgmap v751: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:56:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:44 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:44 smithi186 ceph-mon[33438]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T04:56:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:44.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[29863]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[41141]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T04:56:44.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:56:46.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:45 smithi186 ceph-mon[33438]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T04:56:46.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:45 smithi186 ceph-mon[33438]: pgmap v754: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:56:46.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:45 smithi012 ceph-mon[29863]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T04:56:46.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:45 smithi012 ceph-mon[29863]: pgmap v754: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:56:46.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:45 smithi012 ceph-mon[41141]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T04:56:46.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:45 smithi012 ceph-mon[41141]: pgmap v754: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:56:46.938 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:56:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:56:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:56:47.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:46 smithi186 ceph-mon[33438]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T04:56:47.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:46 smithi012 ceph-mon[29863]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T04:56:47.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:46 smithi012 ceph-mon[41141]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T04:56:48.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:47 smithi186 ceph-mon[33438]: pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-26T04:56:48.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:47 smithi012 ceph-mon[29863]: pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-26T04:56:48.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:47 smithi012 ceph-mon[41141]: pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-26T04:56:49.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:48 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:49.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:48 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:49.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:48 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:56:50.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:49 smithi186 ceph-mon[33438]: pgmap v757: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:56:50.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:49 smithi012 ceph-mon[29863]: pgmap v757: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:56:50.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:49 smithi012 ceph-mon[41141]: pgmap v757: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:56:51.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:50 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:51.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:50 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:51.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:50 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: pgmap v758: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: pgmap v758: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.402 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:52.403 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:51 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: pgmap v758: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:56:52.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:51 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:56:52.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:56:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:56:53.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:52 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi012 to 3249M 2024-03-26T04:56:53.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:52 smithi012 ceph-mon[29863]: Adjusting osd_memory_target on smithi186 to 3248M 2024-03-26T04:56:53.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:52 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:53.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:52 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi012 to 3249M 2024-03-26T04:56:53.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:52 smithi012 ceph-mon[41141]: Adjusting osd_memory_target on smithi186 to 3248M 2024-03-26T04:56:53.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:52 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:53.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:52 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi012 to 3249M 2024-03-26T04:56:53.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:52 smithi186 ceph-mon[33438]: Adjusting osd_memory_target on smithi186 to 3248M 2024-03-26T04:56:53.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:52 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:56:54.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:54 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:54.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:54 smithi012 ceph-mon[29863]: pgmap v759: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1020 B/s wr, 4 op/s 2024-03-26T04:56:54.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:54 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:54.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:54 smithi012 ceph-mon[41141]: pgmap v759: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1020 B/s wr, 4 op/s 2024-03-26T04:56:54.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:54 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:56:54.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:54 smithi186 ceph-mon[33438]: pgmap v759: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1020 B/s wr, 4 op/s 2024-03-26T04:56:56.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:56 smithi012 ceph-mon[29863]: pgmap v760: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:56:56.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:56 smithi012 ceph-mon[41141]: pgmap v760: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:56:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:56 smithi186 ceph-mon[33438]: pgmap v760: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:56:57.025 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:56:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:56:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:56:57.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:57 smithi012 ceph-mon[29863]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T04:56:57.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:57 smithi012 ceph-mon[41141]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T04:56:57.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:57 smithi186 ceph-mon[33438]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T04:56:58.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:56:58 smithi186 ceph-mon[33438]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:56:58.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:56:58 smithi012 ceph-mon[29863]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:56:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:56:58 smithi012 ceph-mon[41141]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:56:59.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:56:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=cleanup t=2024-03-26T04:56:59.02225391Z level=info msg="Completed cleanup jobs" duration=3.160098ms 2024-03-26T04:56:59.458 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 04:56:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=sqlstore.transactions t=2024-03-26T04:56:59.029824397Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:57:00.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:00 smithi012 ceph-mon[29863]: pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:00.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:00 smithi012 ceph-mon[41141]: pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:00.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:00 smithi186 ceph-mon[33438]: pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:02.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:02 smithi012 ceph-mon[29863]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:02.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:02 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]: dispatch 2024-03-26T04:57:02.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:02 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]: dispatch 2024-03-26T04:57:02.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:02 smithi012 ceph-mon[41141]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:02.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:02 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]: dispatch 2024-03-26T04:57:02.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:02 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]: dispatch 2024-03-26T04:57:02.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:02 smithi186 ceph-mon[33438]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:02.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:02 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]: dispatch 2024-03-26T04:57:02.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:02 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]: dispatch 2024-03-26T04:57:02.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:57:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:57:03.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]': finished 2024-03-26T04:57:03.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:03 smithi012 ceph-mon[29863]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T04:57:03.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:03 smithi012 ceph-mon[29863]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T04:57:03.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]': finished 2024-03-26T04:57:03.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:03 smithi012 ceph-mon[41141]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T04:57:03.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:03 smithi012 ceph-mon[41141]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T04:57:03.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.1f", "id": [0, 3]}]': finished 2024-03-26T04:57:03.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:03 smithi186 ceph-mon[33438]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T04:57:03.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:03 smithi186 ceph-mon[33438]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T04:57:04.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:04 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:04.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:04 smithi012 ceph-mon[29863]: pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:04.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:04.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:04 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:04.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:04 smithi012 ceph-mon[41141]: pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:04.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:04.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:04 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:04.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:04 smithi186 ceph-mon[33438]: pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:04.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:05.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:05 smithi012 ceph-mon[29863]: pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T04:57:05.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:05 smithi012 ceph-mon[41141]: pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T04:57:05.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:05 smithi186 ceph-mon[33438]: pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T04:57:06.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-94"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-94"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-94"}]: dispatch 2024-03-26T04:57:06.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-94"}]: dispatch 2024-03-26T04:57:06.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:06.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:06.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-94"}]: dispatch 2024-03-26T04:57:06.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:06 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-94"}]: dispatch 2024-03-26T04:57:07.075 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:57:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:57:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:57:07.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:07 smithi012 ceph-mon[29863]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T04:57:07.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:07 smithi012 ceph-mon[41141]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T04:57:07.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:07 smithi186 ceph-mon[33438]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T04:57:08.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[29863]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T04:57:08.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[29863]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T04:57:08.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:08.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:08.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[41141]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T04:57:08.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[41141]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T04:57:08.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:08.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:08 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:08 smithi186 ceph-mon[33438]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T04:57:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:08 smithi186 ceph-mon[33438]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T04:57:08.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:08.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:08 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:09.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:09.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[29863]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T04:57:09.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:57:09.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[29863]: pgmap v773: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[41141]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T04:57:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:57:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:09 smithi012 ceph-mon[41141]: pgmap v773: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:09.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:09 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:09.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:09 smithi186 ceph-mon[33438]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T04:57:09.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:57:09.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:09 smithi186 ceph-mon[33438]: pgmap v773: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:10.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:10 smithi012 ceph-mon[29863]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T04:57:10.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:10 smithi012 ceph-mon[41141]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T04:57:10.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:10 smithi186 ceph-mon[33438]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T04:57:11.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:11 smithi012 ceph-mon[29863]: pgmap v775: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:11.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:11 smithi012 ceph-mon[41141]: pgmap v775: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:11.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:11 smithi186 ceph-mon[33438]: pgmap v775: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:12.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:57:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:57:14.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:14 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:14.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:14 smithi012 ceph-mon[29863]: pgmap v776: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 869 B/s wr, 2 op/s 2024-03-26T04:57:14.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:14 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:14.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:14 smithi012 ceph-mon[41141]: pgmap v776: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 869 B/s wr, 2 op/s 2024-03-26T04:57:14.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:14 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:14.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:14 smithi186 ceph-mon[33438]: pgmap v776: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 869 B/s wr, 2 op/s 2024-03-26T04:57:16.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:16 smithi012 ceph-mon[29863]: pgmap v777: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 696 B/s rd, 696 B/s wr, 2 op/s 2024-03-26T04:57:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:16 smithi012 ceph-mon[41141]: pgmap v777: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 696 B/s rd, 696 B/s wr, 2 op/s 2024-03-26T04:57:16.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:16 smithi186 ceph-mon[33438]: pgmap v777: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 696 B/s rd, 696 B/s wr, 2 op/s 2024-03-26T04:57:17.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:57:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:57:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:57:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:18 smithi012 ceph-mon[29863]: pgmap v778: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 612 B/s wr, 2 op/s 2024-03-26T04:57:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:18 smithi012 ceph-mon[41141]: pgmap v778: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 612 B/s wr, 2 op/s 2024-03-26T04:57:18.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:18 smithi186 ceph-mon[33438]: pgmap v778: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 612 B/s wr, 2 op/s 2024-03-26T04:57:19.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:19.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:19.899 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 04:57:19 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T04:57:19.648+0000 7fbe1706a700 -1 snap_mapper.add_oid found existing snaps mapped on 117:230edcb8:test-rados-api-smithi012-78824-97::foo:21, removing 2024-03-26T04:57:19.958 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 04:57:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T04:57:19.647+0000 7fe457d54700 -1 snap_mapper.add_oid found existing snaps mapped on 117:230edcb8:test-rados-api-smithi012-78824-97::foo:21, removing 2024-03-26T04:57:19.958 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 04:57:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T04:57:19.648+0000 7f905957b700 -1 snap_mapper.add_oid found existing snaps mapped on 117:230edcb8:test-rados-api-smithi012-78824-97::foo:21, removing 2024-03-26T04:57:20.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[29863]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:57:20.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:20.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-96"}]: dispatch 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-96"}]: dispatch 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[41141]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-96"}]: dispatch 2024-03-26T04:57:20.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:20 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-96"}]: dispatch 2024-03-26T04:57:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:20 smithi186 ceph-mon[33438]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:57:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-96"}]: dispatch 2024-03-26T04:57:20.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:20 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-96"}]: dispatch 2024-03-26T04:57:21.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:21 smithi012 ceph-mon[29863]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T04:57:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:21 smithi012 ceph-mon[41141]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T04:57:21.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:21 smithi186 ceph-mon[33438]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T04:57:22.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[29863]: pgmap v781: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:57:22.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[29863]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T04:57:22.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:22.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[41141]: pgmap v781: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:57:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[41141]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T04:57:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:22.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:22 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:22.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:22 smithi186 ceph-mon[33438]: pgmap v781: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:57:22.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:22 smithi186 ceph-mon[33438]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T04:57:22.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:22.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:22 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:22.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:57:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:57:23.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:23.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[29863]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[29863]: pgmap v784: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[41141]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:23.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:23 smithi012 ceph-mon[41141]: pgmap v784: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:57:23.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:23 smithi186 ceph-mon[33438]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T04:57:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:23 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:23.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:23 smithi186 ceph-mon[33438]: pgmap v784: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:57:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:57:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]: dispatch 2024-03-26T04:57:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[41141]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T04:57:24.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]: dispatch 2024-03-26T04:57:24.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:57:24.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]: dispatch 2024-03-26T04:57:24.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[29863]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T04:57:24.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:24 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]: dispatch 2024-03-26T04:57:24.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:57:24.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]: dispatch 2024-03-26T04:57:24.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:24 smithi186 ceph-mon[33438]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T04:57:24.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:24 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]: dispatch 2024-03-26T04:57:25.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:57:25.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]': finished 2024-03-26T04:57:25.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[29863]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[29863]: pgmap v787: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]': finished 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[41141]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:25.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:25 smithi012 ceph-mon[41141]: pgmap v787: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:25.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:25 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:57:25.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-98", "mode": "writeback"}]': finished 2024-03-26T04:57:25.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:25 smithi186 ceph-mon[33438]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T04:57:25.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:25.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:25.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:25 smithi186 ceph-mon[33438]: pgmap v787: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:26.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]': finished 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[29863]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]': finished 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[41141]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T04:57:26.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:57:26.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-98"}]': finished 2024-03-26T04:57:26.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:57:26.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:26 smithi186 ceph-mon[33438]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T04:57:26.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:57:27.093 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:57:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:57:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:57:27.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:57:27.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[29863]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[29863]: pgmap v790: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[41141]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:57:27.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:27 smithi012 ceph-mon[41141]: pgmap v790: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:27.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:27 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:57:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:57:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:57:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:27 smithi186 ceph-mon[33438]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T04:57:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:27 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:57:27.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:27 smithi186 ceph-mon[33438]: pgmap v790: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:28.096 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T04:57:28.096 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:28.096 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T04:57:28.096 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:57:28.097 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:57:28.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T04:57:28.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:28.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T04:57:28.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:28.359 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:57:28.360 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T04:57:28.360 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:57:28.360 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:57:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:57:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T04:57:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:57:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:57:29.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:29 smithi012 ceph-mon[29863]: pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:29.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T04:57:29.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:29 smithi012 ceph-mon[29863]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T04:57:29.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:29 smithi012 ceph-mon[41141]: pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:29.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T04:57:29.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:29 smithi012 ceph-mon[41141]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T04:57:29.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:29 smithi186 ceph-mon[33438]: pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:57:29.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T04:57:29.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:29 smithi186 ceph-mon[33438]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T04:57:30.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:30.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:30.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:30.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:30.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:30.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:31.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:57:31.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[29863]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T04:57:31.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[29863]: pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[41141]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:31.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:31 smithi012 ceph-mon[41141]: pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:31.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:57:31.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:31 smithi186 ceph-mon[33438]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T04:57:31.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:31.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:31 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]: dispatch 2024-03-26T04:57:31.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:31 smithi186 ceph-mon[33438]: pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:32.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:32 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]': finished 2024-03-26T04:57:32.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:32 smithi012 ceph-mon[29863]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T04:57:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:32 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]': finished 2024-03-26T04:57:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:32 smithi012 ceph-mon[41141]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T04:57:32.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:32 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-98"}]': finished 2024-03-26T04:57:32.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:32 smithi186 ceph-mon[33438]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T04:57:32.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:57:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:57:33.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:33 smithi012 ceph-mon[29863]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T04:57:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:33.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:33 smithi012 ceph-mon[29863]: pgmap v799: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:33.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:33 smithi012 ceph-mon[41141]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T04:57:33.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:33.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:33 smithi012 ceph-mon[41141]: pgmap v799: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:33.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:33 smithi186 ceph-mon[33438]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T04:57:33.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:33.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:33 smithi186 ceph-mon[33438]: pgmap v799: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:34.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[29863]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:57:34.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:34.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:34.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:34.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[41141]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:57:34.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:34.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:34.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:34 smithi186 ceph-mon[33438]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:57:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:34 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:35.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:35.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:57:35.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: pgmap v802: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:57:35.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: pgmap v802: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:57:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:57:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:35.401 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: pgmap v802: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:35.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:37.143 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:57:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:57:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:57:37.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]': finished 2024-03-26T04:57:37.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[41141]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:57:37.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]: dispatch 2024-03-26T04:57:37.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]: dispatch 2024-03-26T04:57:37.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[41141]: pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]': finished 2024-03-26T04:57:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[29863]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:57:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]: dispatch 2024-03-26T04:57:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]: dispatch 2024-03-26T04:57:37.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:37 smithi012 ceph-mon[29863]: pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:37.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-6", "overlaypool": "test-rados-api-smithi012-78824-100"}]': finished 2024-03-26T04:57:37.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:37 smithi186 ceph-mon[33438]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:57:37.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]: dispatch 2024-03-26T04:57:37.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]: dispatch 2024-03-26T04:57:37.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:37 smithi186 ceph-mon[33438]: pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:38.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:38 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:57:38.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]': finished 2024-03-26T04:57:38.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:38 smithi186 ceph-mon[33438]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:57:38.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:57:38.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:38 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:57:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:57:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]': finished 2024-03-26T04:57:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[41141]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:57:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:57:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:57:38.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:57:38.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-100", "mode": "writeback"}]': finished 2024-03-26T04:57:38.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[29863]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:57:38.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:57:38.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:38 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:57:39.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: pgmap v808: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:57:39.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:57:39.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:57:39.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:57:39.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: pgmap v808: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: pgmap v808: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:57:39.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:57:39.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:57:39.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:57:40.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:57:40.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:40 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:57:40.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:40 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:57:40.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:40 smithi186 ceph-mon[33438]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:57:40.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:57:40.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:57:40.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:57:40.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:57:40.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[41141]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:57:40.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:57:40.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:57:40.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:57:40.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:57:40.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[29863]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:57:40.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:57:41.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:57:41.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:41 smithi186 ceph-mon[33438]: pgmap v811: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:41.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T04:57:41.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:41 smithi186 ceph-mon[33438]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:57:41.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:57:41.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[41141]: pgmap v811: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:41.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T04:57:41.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[41141]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:57:41.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:57:41.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[29863]: pgmap v811: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:41.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T04:57:41.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:41 smithi012 ceph-mon[29863]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:57:42.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:42.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:42.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:42.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:42.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:42.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:57:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:57:43.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:57:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:43 smithi186 ceph-mon[33438]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:57:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:43.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:43 smithi186 ceph-mon[33438]: pgmap v814: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:43.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:57:43.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:43.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[29863]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:57:43.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:43.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:43.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[29863]: pgmap v814: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]': finished 2024-03-26T04:57:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[41141]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:57:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]: dispatch 2024-03-26T04:57:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:43.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:43 smithi012 ceph-mon[41141]: pgmap v814: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:44.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:44 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]': finished 2024-03-26T04:57:44.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:44 smithi186 ceph-mon[33438]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:57:44.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:44 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]': finished 2024-03-26T04:57:44.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:44 smithi012 ceph-mon[29863]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:57:44.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:44 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-100"}]': finished 2024-03-26T04:57:44.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:44 smithi012 ceph-mon[41141]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:57:45.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:45 smithi186 ceph-mon[33438]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:57:45.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:45 smithi186 ceph-mon[33438]: pgmap v817: 196 pgs: 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:45.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:45 smithi012 ceph-mon[29863]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:57:45.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:45 smithi012 ceph-mon[29863]: pgmap v817: 196 pgs: 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:45.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:45 smithi012 ceph-mon[41141]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:57:45.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:45 smithi012 ceph-mon[41141]: pgmap v817: 196 pgs: 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:46.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:46 smithi186 ceph-mon[33438]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:57:46.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:46.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:46.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-mon[29863]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:57:46.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:46.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:46.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-mon[41141]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:57:46.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:46.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:47.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:57:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:57:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:57:47.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:47.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[29863]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:57:47.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:57:47.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:57:47.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[29863]: pgmap v820: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[41141]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:57:47.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:47 smithi012 ceph-mon[41141]: pgmap v820: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:47.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:47.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:47 smithi186 ceph-mon[33438]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:57:47.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:57:47.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:57:47.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:57:47.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:47 smithi186 ceph-mon[33438]: pgmap v820: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:48.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:57:48.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[29863]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:57:48.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:57:48.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:57:48.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:57:48.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[41141]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:57:48.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:57:48.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:57:48.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:57:48.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:48 smithi186 ceph-mon[33438]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:57:48.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:57:48.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:57:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:57:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[29863]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[29863]: pgmap v823: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[41141]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:49.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:49 smithi012 ceph-mon[41141]: pgmap v823: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:49.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:57:49.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:49 smithi186 ceph-mon[33438]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:57:49.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:57:49.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:49 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:57:49.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:57:49.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:49 smithi186 ceph-mon[33438]: pgmap v823: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:50.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:57:50.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[29863]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:57:50.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-102"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-102"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[41141]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-102"}]: dispatch 2024-03-26T04:57:50.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:50 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-102"}]: dispatch 2024-03-26T04:57:50.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:57:50.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:50 smithi186 ceph-mon[33438]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:57:50.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:50.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-6"}]: dispatch 2024-03-26T04:57:50.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1921838417' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-102"}]: dispatch 2024-03-26T04:57:50.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:50 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-6", "tierpool": "test-rados-api-smithi012-78824-102"}]: dispatch 2024-03-26T04:57:51.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:51 smithi012 ceph-mon[29863]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:57:51.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:51 smithi012 ceph-mon[29863]: pgmap v826: 196 pgs: 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:51.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:51 smithi012 ceph-mon[41141]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:57:51.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:51 smithi012 ceph-mon[41141]: pgmap v826: 196 pgs: 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:51.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:51 smithi186 ceph-mon[33438]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:57:51.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:51 smithi186 ceph-mon[33438]: pgmap v826: 196 pgs: 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:52.510 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:57:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:57:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:57:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:57:52.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:52 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:57:52.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:57:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:57:53.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:57:53.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[29863]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:57:53.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi012-78824-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:53.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:53.900 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[29863]: pgmap v829: 164 pgs: 164 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:57:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[41141]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:57:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi012-78824-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:53.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:53 smithi012 ceph-mon[41141]: pgmap v829: 164 pgs: 164 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:53.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:57:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:53 smithi186 ceph-mon[33438]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:57:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi012-78824-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:57:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:57:53.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:53 smithi186 ceph-mon[33438]: pgmap v829: 164 pgs: 164 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:54.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:54 smithi012 ceph-mon[29863]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:57:54.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi012-78824-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:57:54.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:54 smithi012 ceph-mon[29863]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:57:54.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:54 smithi012 ceph-mon[41141]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:57:54.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi012-78824-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:57:54.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:54 smithi012 ceph-mon[41141]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:57:54.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:54 smithi186 ceph-mon[33438]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:57:54.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi012-78824-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:57:54.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:54 smithi186 ceph-mon[33438]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:57:56.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:55 smithi012 ceph-mon[29863]: pgmap v832: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:56.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:55 smithi012 ceph-mon[29863]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:57:56.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:55 smithi012 ceph-mon[41141]: pgmap v832: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:56.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:55 smithi012 ceph-mon[41141]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:57:56.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:55 smithi186 ceph-mon[33438]: pgmap v832: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:57:56.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:55 smithi186 ceph-mon[33438]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:57:57.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:57:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:57:57.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:57:57.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[29863]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:57:57.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:57.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:57.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:57:57.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[41141]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:57:57.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:57.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:56 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:57.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:56 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:57:57.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:56 smithi186 ceph-mon[33438]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:57:57.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:57.208 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:56 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:58.109 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:57 smithi186 ceph-mon[33438]: pgmap v835: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:58.109 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:57 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:58.109 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:57 smithi186 ceph-mon[33438]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:57:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:57 smithi012 ceph-mon[29863]: pgmap v835: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:57 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:57 smithi012 ceph-mon[29863]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:57:58.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:57 smithi012 ceph-mon[41141]: pgmap v835: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:58.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:57 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:57:58.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:57 smithi012 ceph-mon[41141]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:57:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[29863]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:57:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[29863]: pgmap v838: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:59.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[41141]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:57:59.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:59.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:59.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:57:59 smithi012 ceph-mon[41141]: pgmap v838: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:57:59.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:59 smithi186 ceph-mon[33438]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:57:59.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:59.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:59 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:57:59.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:57:59 smithi186 ceph-mon[33438]: pgmap v838: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[29863]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[41141]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:58:00.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:00.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:00.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:00.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:00 smithi186 ceph-mon[33438]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:58:00.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:00.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: pgmap v841: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:01.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: pgmap v841: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:01.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:01.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:01.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:58:01.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:01.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: pgmap v841: 236 pgs: 64 unknown, 172 active+clean; 582 KiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:01.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-107", "overlaypool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:01.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:58:01.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:01.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:02.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:02.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:02.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]': finished 2024-03-26T04:58:02.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[29863]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:58:02.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:02.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:02.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]': finished 2024-03-26T04:58:02.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:02 smithi012 ceph-mon[41141]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:58:02.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:02 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:02.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:02 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:02.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-107-cache", "mode": "writeback"}]': finished 2024-03-26T04:58:02.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:02 smithi186 ceph-mon[33438]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:58:02.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:58:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:58:03.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]: dispatch 2024-03-26T04:58:03.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]: dispatch 2024-03-26T04:58:03.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:03.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[29863]: pgmap v844: 236 pgs: 236 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:03.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]: dispatch 2024-03-26T04:58:03.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]: dispatch 2024-03-26T04:58:03.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:03.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:03 smithi012 ceph-mon[41141]: pgmap v844: 236 pgs: 236 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:03.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]: dispatch 2024-03-26T04:58:03.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]: dispatch 2024-03-26T04:58:03.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:03.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:03 smithi186 ceph-mon[33438]: pgmap v844: 236 pgs: 236 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]': finished 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[29863]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:04.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]': finished 2024-03-26T04:58:04.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[41141]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:58:04.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:04.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:04.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:04.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:04.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-107"}]': finished 2024-03-26T04:58:04.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:04 smithi186 ceph-mon[33438]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:58:04.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/472721437' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:04.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]: dispatch 2024-03-26T04:58:04.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:04 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:04.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:05.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:05.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:05.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[29863]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:58:05.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[29863]: pgmap v847: 236 pgs: 236 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:05.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:05.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:05.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[41141]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:58:05.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:05 smithi012 ceph-mon[41141]: pgmap v847: 236 pgs: 236 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:05.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:05 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:05.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-107", "tierpool": "test-rados-api-smithi012-78824-107-cache"}]': finished 2024-03-26T04:58:05.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:05 smithi186 ceph-mon[33438]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:58:05.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:05 smithi186 ceph-mon[33438]: pgmap v847: 236 pgs: 236 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:06.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:06 smithi012 ceph-mon[29863]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:58:06.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:06 smithi012 ceph-mon[41141]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:58:06.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:06 smithi186 ceph-mon[33438]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:58:07.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:58:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:58:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:58:07.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[29863]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:58:07.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:07.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:07.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:58:07.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[29863]: pgmap v850: 172 pgs: 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:07.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[41141]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:58:07.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:07.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:07.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:58:07.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:07 smithi012 ceph-mon[41141]: pgmap v850: 172 pgs: 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:07.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:07 smithi186 ceph-mon[33438]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:58:07.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:07.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:07.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:58:07.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:07 smithi186 ceph-mon[33438]: pgmap v850: 172 pgs: 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:08.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:58:08.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:08 smithi012 ceph-mon[29863]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:58:08.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:08.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:58:08.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:08 smithi012 ceph-mon[41141]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:58:08.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:08.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:58:08.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:08 smithi186 ceph-mon[33438]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:58:08.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:09.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[29863]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:58:09.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[29863]: pgmap v853: 172 pgs: 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:09.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:09.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[29863]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:58:09.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[41141]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:58:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[41141]: pgmap v853: 172 pgs: 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:09.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:09 smithi012 ceph-mon[41141]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:58:09.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:09 smithi186 ceph-mon[33438]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:58:09.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:09 smithi186 ceph-mon[33438]: pgmap v853: 172 pgs: 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:09.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:09.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:09 smithi186 ceph-mon[33438]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:58:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:11 smithi012 ceph-mon[29863]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:58:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:11 smithi012 ceph-mon[29863]: pgmap v856: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:11 smithi012 ceph-mon[41141]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:58:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:11 smithi012 ceph-mon[41141]: pgmap v856: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:11 smithi186 ceph-mon[33438]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:58:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:11 smithi186 ceph-mon[33438]: pgmap v856: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[41141]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[41141]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[29863]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[29863]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:58:12.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:12.707 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:58:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:58:12.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:12 smithi186 ceph-mon[33438]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:12.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:12.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:12 smithi186 ceph-mon[33438]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:58:12.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:13.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:13.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[29863]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:58:13.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-109", "overlaypool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:13.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:13.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[29863]: pgmap v859: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:13.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:13.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[41141]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:58:13.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-109", "overlaypool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:13.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:13.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:13 smithi012 ceph-mon[41141]: pgmap v859: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:13.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:13.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:13 smithi186 ceph-mon[33438]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:58:13.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-109", "overlaypool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:13.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:13.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:13 smithi186 ceph-mon[33438]: pgmap v859: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-109", "overlaypool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[41141]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-109-cache", "mode": "writeback"}]': finished 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[41141]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-109", "overlaypool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:14.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[29863]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:58:14.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:14.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:14.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-109-cache", "mode": "writeback"}]': finished 2024-03-26T04:58:14.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:14 smithi012 ceph-mon[29863]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:58:14.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-109", "overlaypool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:14.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:14 smithi186 ceph-mon[33438]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:58:14.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:58:14.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:14 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:14.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-109-cache", "mode": "writeback"}]': finished 2024-03-26T04:58:14.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:14 smithi186 ceph-mon[33438]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[29863]: pgmap v862: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[29863]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:58:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[41141]: pgmap v862: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:15.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:58:15.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[41141]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:58:15.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:58:15.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:58:15.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:15 smithi186 ceph-mon[33438]: pgmap v862: 212 pgs: 32 creating+peering, 180 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:15.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:58:15.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:15 smithi186 ceph-mon[33438]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:58:15.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:58:17.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:58:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:58:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:58:17.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:58:17.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[29863]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:58:17.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:58:17.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[29863]: pgmap v865: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:17.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:58:17.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[41141]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:58:17.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:58:17.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:17 smithi012 ceph-mon[41141]: pgmap v865: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:17.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:58:17.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:17 smithi186 ceph-mon[33438]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:58:17.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:58:17.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:17 smithi186 ceph-mon[33438]: pgmap v865: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[29863]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:18.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:58:18.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[41141]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:58:18.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T04:58:18.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:18 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:18.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:18 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:18.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:58:18.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:18 smithi186 ceph-mon[33438]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:58:18.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T04:58:18.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:18 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:19.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T04:58:19.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[29863]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T04:58:19.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:19.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:19.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:19.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[29863]: pgmap v868: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:19.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T04:58:19.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[41141]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T04:58:19.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:19.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:19.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:19.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:19 smithi012 ceph-mon[41141]: pgmap v868: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:19.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T04:58:19.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:19 smithi186 ceph-mon[33438]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T04:58:19.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:19.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:19.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:19.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:19 smithi186 ceph-mon[33438]: pgmap v868: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:20.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:20.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:20 smithi012 ceph-mon[29863]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T04:58:20.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:20.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:20.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:20 smithi012 ceph-mon[41141]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T04:58:20.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:20.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:20.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:20 smithi186 ceph-mon[33438]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T04:58:20.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]: dispatch 2024-03-26T04:58:21.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:21.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[29863]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T04:58:21.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[29863]: pgmap v871: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:21.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[29863]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T04:58:21.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:21.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[41141]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T04:58:21.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[41141]: pgmap v871: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:21.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:21 smithi012 ceph-mon[41141]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T04:58:21.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-109", "tierpool": "test-rados-api-smithi012-78824-109-cache"}]': finished 2024-03-26T04:58:21.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:21 smithi186 ceph-mon[33438]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T04:58:21.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:21 smithi186 ceph-mon[33438]: pgmap v871: 212 pgs: 212 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:21.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:21 smithi186 ceph-mon[33438]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T04:58:22.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:58:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:58:23.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[29863]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T04:58:23.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:23.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:23.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[29863]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:23.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[29863]: pgmap v874: 172 pgs: 172 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:23.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[41141]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T04:58:23.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:23.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:23.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[41141]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:23.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:23 smithi012 ceph-mon[41141]: pgmap v874: 172 pgs: 172 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:23.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:23 smithi186 ceph-mon[33438]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T04:58:23.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:23.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:23.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:23 smithi186 ceph-mon[33438]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:23.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:23 smithi186 ceph-mon[33438]: pgmap v874: 172 pgs: 172 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:24.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:24.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:24 smithi186 ceph-mon[33438]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T04:58:24.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:24.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:24.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:24 smithi012 ceph-mon[29863]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T04:58:24.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:24.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:24.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:24 smithi012 ceph-mon[41141]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T04:58:24.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]: dispatch 2024-03-26T04:58:25.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:25.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:25 smithi186 ceph-mon[33438]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T04:58:25.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:25 smithi186 ceph-mon[33438]: pgmap v877: 172 pgs: 172 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:25.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:25.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:25 smithi012 ceph-mon[29863]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T04:58:25.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:25 smithi012 ceph-mon[29863]: pgmap v877: 172 pgs: 172 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:25.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2118291698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-109"}]': finished 2024-03-26T04:58:25.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:25 smithi012 ceph-mon[41141]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T04:58:25.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:25 smithi012 ceph-mon[41141]: pgmap v877: 172 pgs: 172 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:26.676 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[29863]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T04:58:26.676 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:58:26.676 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[29863]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[41141]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[41141]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T04:58:26.677 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:58:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:26 smithi186 ceph-mon[33438]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T04:58:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:58:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:58:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:26 smithi186 ceph-mon[33438]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T04:58:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]: dispatch 2024-03-26T04:58:27.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:58:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:58:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:58:27.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:27 smithi186 ceph-mon[33438]: pgmap v880: 164 pgs: 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:58:27.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:58:27.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:27 smithi186 ceph-mon[33438]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T04:58:27.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:27 smithi012 ceph-mon[29863]: pgmap v880: 164 pgs: 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:58:27.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:58:27.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:27 smithi012 ceph-mon[29863]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T04:58:27.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:27 smithi012 ceph-mon[41141]: pgmap v880: 164 pgs: 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:58:27.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1293787669' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi012-78824-104"}]': finished 2024-03-26T04:58:27.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:27 smithi012 ceph-mon[41141]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T04:58:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:58:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:28 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:58:28.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:28 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:29.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:58:29.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:29 smithi186 ceph-mon[33438]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T04:58:29.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:29.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:29 smithi186 ceph-mon[33438]: pgmap v883: 164 pgs: 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:58:30.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:58:30.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[29863]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T04:58:30.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:30.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[29863]: pgmap v883: 164 pgs: 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:58:30.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi012-78824-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:58:30.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[41141]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T04:58:30.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:30.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:29 smithi012 ceph-mon[41141]: pgmap v883: 164 pgs: 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:58:30.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:30 smithi186 ceph-mon[33438]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T04:58:30.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:30.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:30 smithi186 ceph-mon[33438]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T04:58:31.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:30 smithi012 ceph-mon[29863]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T04:58:31.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:31.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:30 smithi012 ceph-mon[29863]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T04:58:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:30 smithi012 ceph-mon[41141]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T04:58:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi012-78824-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:30 smithi012 ceph-mon[41141]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T04:58:31.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:31 smithi186 ceph-mon[33438]: pgmap v886: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:31.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:31 smithi186 ceph-mon[33438]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T04:58:31.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:31 smithi012 ceph-mon[29863]: pgmap v886: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:31 smithi012 ceph-mon[29863]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T04:58:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:31 smithi012 ceph-mon[41141]: pgmap v886: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:31 smithi012 ceph-mon[41141]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T04:58:32.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:32.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:58:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:58:33.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:33.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:33 smithi186 ceph-mon[33438]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T04:58:33.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:33.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:33.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:33 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:33.708 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:33 smithi186 ceph-mon[33438]: pgmap v889: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[29863]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[29863]: pgmap v889: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[41141]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:33.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:33.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:33.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:33 smithi012 ceph-mon[41141]: pgmap v889: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:34.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:34.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:34 smithi186 ceph-mon[33438]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T04:58:34.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-112"}]: dispatch 2024-03-26T04:58:34.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[29863]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-112"}]: dispatch 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[41141]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T04:58:34.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-112"}]: dispatch 2024-03-26T04:58:34.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:35.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-112"}]': finished 2024-03-26T04:58:35.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[29863]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T04:58:35.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:35.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[29863]: pgmap v892: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:35.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-112"}]': finished 2024-03-26T04:58:35.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[41141]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T04:58:35.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:35.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:35 smithi012 ceph-mon[41141]: pgmap v892: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:35.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-112"}]': finished 2024-03-26T04:58:35.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:35 smithi186 ceph-mon[33438]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T04:58:35.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:35.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:35 smithi186 ceph-mon[33438]: pgmap v892: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:36.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:36.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-mon[29863]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T04:58:36.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112"}]: dispatch 2024-03-26T04:58:36.899 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:58:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:58:36.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:36.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-mon[41141]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T04:58:36.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112"}]: dispatch 2024-03-26T04:58:36.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:36.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:36 smithi186 ceph-mon[33438]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T04:58:36.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112"}]: dispatch 2024-03-26T04:58:37.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112"}]': finished 2024-03-26T04:58:37.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:37 smithi012 ceph-mon[29863]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T04:58:37.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:37 smithi012 ceph-mon[29863]: pgmap v895: 204 pgs: 204 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:58:37.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112"}]': finished 2024-03-26T04:58:37.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:37 smithi012 ceph-mon[41141]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T04:58:37.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:37 smithi012 ceph-mon[41141]: pgmap v895: 204 pgs: 204 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:58:37.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-112"}]': finished 2024-03-26T04:58:37.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:37 smithi186 ceph-mon[33438]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T04:58:37.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:37 smithi186 ceph-mon[33438]: pgmap v895: 204 pgs: 204 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:58:38.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:38 smithi012 ceph-mon[29863]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T04:58:38.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:38 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:38.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:38 smithi012 ceph-mon[41141]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T04:58:38.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:38 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:38.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:38 smithi186 ceph-mon[33438]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T04:58:38.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:38 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[29863]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[29863]: pgmap v898: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[41141]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[41141]: pgmap v898: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:39.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:39.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:39 smithi186 ceph-mon[33438]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T04:58:39.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:39.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:39 smithi186 ceph-mon[33438]: pgmap v898: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:39.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:40.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[29863]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T04:58:40.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:40.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:40.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[29863]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T04:58:40.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-114"}]: dispatch 2024-03-26T04:58:40.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[41141]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T04:58:40.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:40.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:40.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[41141]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T04:58:40.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-114"}]: dispatch 2024-03-26T04:58:40.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:40 smithi186 ceph-mon[33438]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T04:58:40.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:40.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:40.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:40 smithi186 ceph-mon[33438]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T04:58:40.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-114"}]: dispatch 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[29863]: pgmap v901: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-114"}]': finished 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[29863]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-114", "mode": "writeback"}]: dispatch 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[41141]: pgmap v901: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-114"}]': finished 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[41141]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T04:58:41.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-114", "mode": "writeback"}]: dispatch 2024-03-26T04:58:41.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:41 smithi186 ceph-mon[33438]: pgmap v901: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:41.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-114"}]': finished 2024-03-26T04:58:41.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:41 smithi186 ceph-mon[33438]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T04:58:41.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-114", "mode": "writeback"}]: dispatch 2024-03-26T04:58:42.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:42 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:42.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-114", "mode": "writeback"}]': finished 2024-03-26T04:58:42.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:42 smithi012 ceph-mon[41141]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T04:58:42.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:42 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:42.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-114", "mode": "writeback"}]': finished 2024-03-26T04:58:42.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:42 smithi012 ceph-mon[29863]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T04:58:42.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:58:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:58:42.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:42 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:42.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-114", "mode": "writeback"}]': finished 2024-03-26T04:58:42.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:42 smithi186 ceph-mon[33438]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T04:58:43.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:43.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:43.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:43.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[29863]: pgmap v904: 204 pgs: 204 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:43.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:43.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:43.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:43.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:43 smithi012 ceph-mon[41141]: pgmap v904: 204 pgs: 204 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:43.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:58:43.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:43.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:43 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:43.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:43 smithi186 ceph-mon[33438]: pgmap v904: 204 pgs: 204 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:44.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:44.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:44 smithi012 ceph-mon[29863]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T04:58:44.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114"}]: dispatch 2024-03-26T04:58:44.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:44.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:44 smithi012 ceph-mon[41141]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T04:58:44.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114"}]: dispatch 2024-03-26T04:58:44.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:58:44.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:44 smithi186 ceph-mon[33438]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T04:58:44.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114"}]: dispatch 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114"}]': finished 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[29863]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[29863]: pgmap v907: 204 pgs: 204 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114"}]': finished 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[41141]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T04:58:45.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:45 smithi012 ceph-mon[41141]: pgmap v907: 204 pgs: 204 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:45.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:45 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:58:45.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-114"}]': finished 2024-03-26T04:58:45.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:45 smithi186 ceph-mon[33438]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T04:58:45.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:45 smithi186 ceph-mon[33438]: pgmap v907: 204 pgs: 204 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:58:46.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:46 smithi012 ceph-mon[29863]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T04:58:46.899 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:58:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:58:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:58:46.901 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:46 smithi012 ceph-mon[41141]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T04:58:46.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:46 smithi186 ceph-mon[33438]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T04:58:47.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[29863]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T04:58:47.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:47.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[29863]: pgmap v910: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:47.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:47.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[29863]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T04:58:47.801 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[41141]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T04:58:47.801 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:47.801 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[41141]: pgmap v910: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:47.801 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:47.801 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:47 smithi012 ceph-mon[41141]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T04:58:47.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:47 smithi186 ceph-mon[33438]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T04:58:47.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:58:47.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:47 smithi186 ceph-mon[33438]: pgmap v910: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:47.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:58:47.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:47 smithi186 ceph-mon[33438]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T04:58:48.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:48 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:48.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:48 smithi012 ceph-mon[29863]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T04:58:48.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:48 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:48.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:48 smithi012 ceph-mon[41141]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T04:58:48.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:48 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:48.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:48 smithi186 ceph-mon[33438]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T04:58:49.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:49.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:49.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:49 smithi012 ceph-mon[29863]: pgmap v913: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:49.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:49.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:49.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:49 smithi012 ceph-mon[41141]: pgmap v913: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:49.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:58:49.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:58:49.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:49 smithi186 ceph-mon[33438]: pgmap v913: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:58:50.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:50.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:50 smithi012 ceph-mon[29863]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T04:58:50.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-116"}]: dispatch 2024-03-26T04:58:50.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:50.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:50 smithi012 ceph-mon[41141]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T04:58:50.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-116"}]: dispatch 2024-03-26T04:58:50.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:58:50.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:50 smithi186 ceph-mon[33438]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T04:58:50.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-116"}]: dispatch 2024-03-26T04:58:51.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-116"}]': finished 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[29863]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-116", "mode": "writeback"}]: dispatch 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[29863]: pgmap v916: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-116", "mode": "writeback"}]': finished 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-116"}]': finished 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[41141]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-116", "mode": "writeback"}]: dispatch 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[41141]: pgmap v916: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:51.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:51.900 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-116", "mode": "writeback"}]': finished 2024-03-26T04:58:51.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-116"}]': finished 2024-03-26T04:58:51.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:51 smithi186 ceph-mon[33438]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T04:58:51.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-116", "mode": "writeback"}]: dispatch 2024-03-26T04:58:51.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:51 smithi186 ceph-mon[33438]: pgmap v916: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:58:51.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:51 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:58:51.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-116", "mode": "writeback"}]': finished 2024-03-26T04:58:52.835 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[41141]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T04:58:52.835 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-03-26T04:58:52.835 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[41141]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-03-26T04:58:52.835 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[41141]: 326.5 scrub starts 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[41141]: 326.5 scrub ok 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[29863]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[29863]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[29863]: 326.5 scrub starts 2024-03-26T04:58:52.836 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:52 smithi012 ceph-mon[29863]: 326.5 scrub ok 2024-03-26T04:58:52.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-mon[33438]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T04:58:52.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-03-26T04:58:52.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-mon[33438]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.5"}]: dispatch 2024-03-26T04:58:52.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:58:52.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-mon[33438]: 326.5 scrub starts 2024-03-26T04:58:52.928 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-mon[33438]: 326.5 scrub ok 2024-03-26T04:58:52.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:58:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:58:53.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:53.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:53 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:53.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:53 smithi012 ceph-mon[29863]: pgmap v918: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-26T04:58:53.862 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:53.862 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:53 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:53.862 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:53 smithi012 ceph-mon[41141]: pgmap v918: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-26T04:58:53.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:58:53.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:53 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:58:53.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:53 smithi186 ceph-mon[33438]: pgmap v918: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-26T04:58:56.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-mon[29863]: pgmap v919: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:58:56.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:56.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:56.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-mon[41141]: pgmap v919: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:58:56.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:56.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:56 smithi186 ceph-mon[33438]: pgmap v919: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:58:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:56 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:56 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:58:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:58:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:58:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:57 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:57 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:58:57 smithi186 ceph-mon[33438]: pgmap v920: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 965 B/s wr, 2 op/s 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[29863]: pgmap v920: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 965 B/s wr, 2 op/s 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:58:57.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:58:57.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:58:57.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:58:57 smithi012 ceph-mon[41141]: pgmap v920: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 965 B/s wr, 2 op/s 2024-03-26T04:59:00.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:00 smithi012 ceph-mon[29863]: pgmap v921: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 851 B/s wr, 2 op/s 2024-03-26T04:59:00.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:00 smithi012 ceph-mon[41141]: pgmap v921: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 851 B/s wr, 2 op/s 2024-03-26T04:59:00.430 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:00 smithi186 ceph-mon[33438]: pgmap v921: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 851 B/s wr, 2 op/s 2024-03-26T04:59:02.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:02 smithi012 ceph-mon[29863]: pgmap v922: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-03-26T04:59:02.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:02 smithi012 ceph-mon[41141]: pgmap v922: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-03-26T04:59:02.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:02 smithi186 ceph-mon[33438]: pgmap v922: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-03-26T04:59:02.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:59:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:59:04.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[29863]: pgmap v923: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[41141]: pgmap v923: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:04.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:04.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:04 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:04.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:04 smithi186 ceph-mon[33438]: pgmap v923: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:59:04.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:04 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:04.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:06.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:06 smithi012 ceph-mon[29863]: pgmap v924: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T04:59:06.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:06 smithi012 ceph-mon[41141]: pgmap v924: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T04:59:06.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:06 smithi186 ceph-mon[33438]: pgmap v924: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T04:59:07.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:59:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:59:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:59:08.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:08 smithi012 ceph-mon[29863]: pgmap v925: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:59:08.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:08 smithi012 ceph-mon[29863]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T04:59:08.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:08.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:08 smithi012 ceph-mon[41141]: pgmap v925: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:59:08.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:08 smithi012 ceph-mon[41141]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T04:59:08.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:08 smithi186 ceph-mon[33438]: pgmap v925: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:59:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:08 smithi186 ceph-mon[33438]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T04:59:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[29863]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116"}]: dispatch 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[29863]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[41141]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116"}]: dispatch 2024-03-26T04:59:09.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:09 smithi012 ceph-mon[41141]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:09.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:09.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:09 smithi186 ceph-mon[33438]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T04:59:09.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116"}]: dispatch 2024-03-26T04:59:09.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:09 smithi186 ceph-mon[33438]: pgmap v928: 204 pgs: 204 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:10.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:10 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:10.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116"}]': finished 2024-03-26T04:59:10.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:10 smithi012 ceph-mon[29863]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T04:59:10.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:10 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:10.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116"}]': finished 2024-03-26T04:59:10.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:10 smithi012 ceph-mon[41141]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T04:59:10.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:10 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:10.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-116"}]': finished 2024-03-26T04:59:10.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:10 smithi186 ceph-mon[33438]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[29863]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[29863]: pgmap v931: 172 pgs: 172 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[29863]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[41141]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[41141]: pgmap v931: 172 pgs: 172 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[41141]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T04:59:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:11 smithi186 ceph-mon[33438]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T04:59:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:11 smithi186 ceph-mon[33438]: pgmap v931: 172 pgs: 172 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:11 smithi186 ceph-mon[33438]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T04:59:11.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:12.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:59:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:59:13.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[29863]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[29863]: pgmap v934: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[41141]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:13.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:13 smithi012 ceph-mon[41141]: pgmap v934: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:59:13.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:13.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:13 smithi186 ceph-mon[33438]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T04:59:13.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:13.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:13 smithi186 ceph-mon[33438]: pgmap v934: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:59:14.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:14 smithi012 ceph-mon[29863]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T04:59:14.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:14.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:14 smithi012 ceph-mon[41141]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T04:59:14.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:14.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:14 smithi186 ceph-mon[33438]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T04:59:14.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[29863]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-118"}]: dispatch 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[29863]: pgmap v937: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-118"}]': finished 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[41141]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-118"}]: dispatch 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[41141]: pgmap v937: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-118"}]': finished 2024-03-26T04:59:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:15 smithi186 ceph-mon[33438]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T04:59:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-118"}]: dispatch 2024-03-26T04:59:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:15 smithi186 ceph-mon[33438]: pgmap v937: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-118"}]': finished 2024-03-26T04:59:16.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[29863]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-118", "mode": "writeback"}]: dispatch 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-118", "mode": "writeback"}]': finished 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[29863]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[41141]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-118", "mode": "writeback"}]: dispatch 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-118", "mode": "writeback"}]': finished 2024-03-26T04:59:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-mon[41141]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T04:59:16.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:16 smithi186 ceph-mon[33438]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T04:59:16.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-118", "mode": "writeback"}]: dispatch 2024-03-26T04:59:16.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:16 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:16.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-118", "mode": "writeback"}]': finished 2024-03-26T04:59:16.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:16 smithi186 ceph-mon[33438]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T04:59:17.123 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:59:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:59:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:59:17.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:17 smithi012 ceph-mon[29863]: pgmap v940: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:59:17.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:17 smithi012 ceph-mon[29863]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T04:59:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:17 smithi012 ceph-mon[41141]: pgmap v940: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:59:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:17 smithi012 ceph-mon[41141]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T04:59:17.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:17 smithi186 ceph-mon[33438]: pgmap v940: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:59:17.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:17 smithi186 ceph-mon[33438]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T04:59:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[29863]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T04:59:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118"}]: dispatch 2024-03-26T04:59:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:18.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[41141]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T04:59:18.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118"}]: dispatch 2024-03-26T04:59:18.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:18.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:18.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:18 smithi186 ceph-mon[33438]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T04:59:18.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118"}]: dispatch 2024-03-26T04:59:19.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:19.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118"}]': finished 2024-03-26T04:59:19.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[29863]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T04:59:19.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:19.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:19.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[29863]: pgmap v944: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 525 B/s wr, 1 op/s 2024-03-26T04:59:19.383 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:19.383 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118"}]': finished 2024-03-26T04:59:19.383 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[41141]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T04:59:19.383 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:19.383 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:19.383 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:19 smithi012 ceph-mon[41141]: pgmap v944: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 525 B/s wr, 1 op/s 2024-03-26T04:59:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:19 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-118"}]': finished 2024-03-26T04:59:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:19 smithi186 ceph-mon[33438]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T04:59:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:19 smithi186 ceph-mon[33438]: pgmap v944: 204 pgs: 204 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 525 B/s wr, 1 op/s 2024-03-26T04:59:21.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:20 smithi012 ceph-mon[29863]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T04:59:21.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:20 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:21.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:20 smithi012 ceph-mon[41141]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T04:59:21.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:20 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:21.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:20 smithi186 ceph-mon[33438]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T04:59:21.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:20 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:22.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:21 smithi012 ceph-mon[29863]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T04:59:22.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:22.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:21 smithi012 ceph-mon[29863]: pgmap v947: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:22.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:21 smithi012 ceph-mon[41141]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T04:59:22.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:22.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:21 smithi012 ceph-mon[41141]: pgmap v947: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:21 smithi186 ceph-mon[33438]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T04:59:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:21 smithi186 ceph-mon[33438]: pgmap v947: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:22.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:22.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:22 smithi186 ceph-mon[33438]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T04:59:22.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:22.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:59:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:59:23.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:23.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:22 smithi012 ceph-mon[29863]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T04:59:23.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:23.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:23.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:22 smithi012 ceph-mon[41141]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T04:59:23.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:24.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:24.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:24.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[29863]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T04:59:24.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-120"}]: dispatch 2024-03-26T04:59:24.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[29863]: pgmap v950: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[41141]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T04:59:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-120"}]: dispatch 2024-03-26T04:59:24.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:23 smithi012 ceph-mon[41141]: pgmap v950: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:23 smithi186 ceph-mon[33438]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T04:59:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-120"}]: dispatch 2024-03-26T04:59:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:23 smithi186 ceph-mon[33438]: pgmap v950: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:25.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-120"}]': finished 2024-03-26T04:59:25.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:24 smithi012 ceph-mon[29863]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T04:59:25.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-120", "mode": "writeback"}]: dispatch 2024-03-26T04:59:25.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-120"}]': finished 2024-03-26T04:59:25.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:24 smithi012 ceph-mon[41141]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T04:59:25.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-120", "mode": "writeback"}]: dispatch 2024-03-26T04:59:25.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-120"}]': finished 2024-03-26T04:59:25.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:24 smithi186 ceph-mon[33438]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T04:59:25.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-120", "mode": "writeback"}]: dispatch 2024-03-26T04:59:26.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-120", "mode": "writeback"}]': finished 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[29863]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[29863]: pgmap v953: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-120", "mode": "writeback"}]': finished 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[41141]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T04:59:26.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:26.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:25 smithi012 ceph-mon[41141]: pgmap v953: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:25 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-120", "mode": "writeback"}]': finished 2024-03-26T04:59:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:25 smithi186 ceph-mon[33438]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T04:59:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:25 smithi186 ceph-mon[33438]: pgmap v953: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:27.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:27.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-mon[29863]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T04:59:27.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120"}]: dispatch 2024-03-26T04:59:27.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:27.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-mon[41141]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T04:59:27.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120"}]: dispatch 2024-03-26T04:59:27.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:59:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:59:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:59:27.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:27.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:26 smithi186 ceph-mon[33438]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T04:59:27.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120"}]: dispatch 2024-03-26T04:59:28.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120"}]': finished 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[29863]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[29863]: pgmap v956: 204 pgs: 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[29863]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120"}]': finished 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[41141]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[41141]: pgmap v956: 204 pgs: 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:59:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:28.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:28 smithi012 ceph-mon[41141]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T04:59:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:28 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-120"}]': finished 2024-03-26T04:59:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:28 smithi186 ceph-mon[33438]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T04:59:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:28 smithi186 ceph-mon[33438]: pgmap v956: 204 pgs: 204 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:59:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:28 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:28 smithi186 ceph-mon[33438]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T04:59:30.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:29 smithi012 ceph-mon[29863]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T04:59:30.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:30.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:29 smithi012 ceph-mon[29863]: pgmap v959: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:59:30.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:29 smithi012 ceph-mon[41141]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T04:59:30.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:30.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:29 smithi012 ceph-mon[41141]: pgmap v959: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:59:30.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:29 smithi186 ceph-mon[33438]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T04:59:30.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:30.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:29 smithi186 ceph-mon[33438]: pgmap v959: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:59:31.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:31.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:30 smithi012 ceph-mon[29863]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T04:59:31.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:30 smithi012 ceph-mon[41141]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T04:59:31.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:31.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:31.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:30 smithi186 ceph-mon[33438]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T04:59:31.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:32.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[29863]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-122"}]: dispatch 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[29863]: pgmap v962: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-122"}]': finished 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[41141]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-122"}]: dispatch 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[41141]: pgmap v962: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:32.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-122"}]': finished 2024-03-26T04:59:32.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:32.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:31 smithi186 ceph-mon[33438]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T04:59:32.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-122"}]: dispatch 2024-03-26T04:59:32.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:31 smithi186 ceph-mon[33438]: pgmap v962: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:32.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-122"}]': finished 2024-03-26T04:59:32.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:59:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:59:32.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:32 smithi186 ceph-mon[33438]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T04:59:32.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-122", "mode": "writeback"}]: dispatch 2024-03-26T04:59:32.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:32 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:32.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-122", "mode": "writeback"}]': finished 2024-03-26T04:59:32.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:32 smithi186 ceph-mon[33438]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T04:59:33.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[29863]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T04:59:33.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-122", "mode": "writeback"}]: dispatch 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-122", "mode": "writeback"}]': finished 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[29863]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[41141]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-122", "mode": "writeback"}]: dispatch 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-122", "mode": "writeback"}]': finished 2024-03-26T04:59:33.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:32 smithi012 ceph-mon[41141]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T04:59:34.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:34.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:34.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:33 smithi012 ceph-mon[29863]: pgmap v965: 204 pgs: 204 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:34.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:34.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:34.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:33 smithi012 ceph-mon[41141]: pgmap v965: 204 pgs: 204 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:34.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:34.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:34.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:33 smithi186 ceph-mon[33438]: pgmap v965: 204 pgs: 204 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[29863]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122"}]: dispatch 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[41141]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122"}]: dispatch 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:35.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:35.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:35.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:34 smithi186 ceph-mon[33438]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T04:59:35.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122"}]: dispatch 2024-03-26T04:59:35.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:34 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:35.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:36.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:36.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122"}]': finished 2024-03-26T04:59:36.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[41141]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T04:59:36.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[41141]: pgmap v968: 204 pgs: 204 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:36.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:36.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122"}]': finished 2024-03-26T04:59:36.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[29863]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T04:59:36.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:35 smithi012 ceph-mon[29863]: pgmap v968: 204 pgs: 204 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:36.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:35 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:36.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-122"}]': finished 2024-03-26T04:59:36.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:35 smithi186 ceph-mon[33438]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T04:59:36.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:35 smithi186 ceph-mon[33438]: pgmap v968: 204 pgs: 204 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:37.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:36 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:37.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:36 smithi012 ceph-mon[29863]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T04:59:37.149 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:59:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:59:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:59:37.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:36 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:37.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:36 smithi012 ceph-mon[41141]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T04:59:37.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:36 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:37.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:36 smithi186 ceph-mon[33438]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T04:59:38.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:37 smithi012 ceph-mon[29863]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T04:59:38.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:38.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:37 smithi012 ceph-mon[29863]: pgmap v971: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:38.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:37 smithi012 ceph-mon[41141]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T04:59:38.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:38.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:37 smithi012 ceph-mon[41141]: pgmap v971: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:38.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:37 smithi186 ceph-mon[33438]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T04:59:38.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:38.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:37 smithi186 ceph-mon[33438]: pgmap v971: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:39.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:39.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:38 smithi012 ceph-mon[29863]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T04:59:39.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:39.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:38 smithi012 ceph-mon[41141]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T04:59:39.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:39.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:38 smithi186 ceph-mon[33438]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T04:59:40.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:39 smithi012 ceph-mon[29863]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T04:59:40.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:40.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:39 smithi012 ceph-mon[29863]: pgmap v974: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:39 smithi012 ceph-mon[41141]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T04:59:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:39 smithi012 ceph-mon[41141]: pgmap v974: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:40.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:39 smithi186 ceph-mon[33438]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T04:59:40.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:40.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:39 smithi186 ceph-mon[33438]: pgmap v974: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:41.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:41.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:40 smithi012 ceph-mon[29863]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T04:59:41.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-124"}]: dispatch 2024-03-26T04:59:41.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:41.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:40 smithi012 ceph-mon[41141]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T04:59:41.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-124"}]: dispatch 2024-03-26T04:59:41.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:41.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:40 smithi186 ceph-mon[33438]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T04:59:41.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-124"}]: dispatch 2024-03-26T04:59:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-124"}]': finished 2024-03-26T04:59:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-124", "mode": "writeback"}]: dispatch 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: pgmap v977: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-124", "mode": "writeback"}]': finished 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[41141]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-124"}]': finished 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-124", "mode": "writeback"}]: dispatch 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: pgmap v977: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-124", "mode": "writeback"}]': finished 2024-03-26T04:59:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:41 smithi012 ceph-mon[29863]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-124"}]': finished 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-124", "mode": "writeback"}]: dispatch 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: pgmap v977: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-124", "mode": "writeback"}]': finished 2024-03-26T04:59:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:41 smithi186 ceph-mon[33438]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T04:59:42.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:42 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:42.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:59:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:59:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:42 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:43.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:42 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[29863]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[29863]: pgmap v980: 204 pgs: 204 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[41141]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:43 smithi012 ceph-mon[41141]: pgmap v980: 204 pgs: 204 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:59:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:43 smithi186 ceph-mon[33438]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T04:59:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:43 smithi186 ceph-mon[33438]: pgmap v980: 204 pgs: 204 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:59:45.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:45.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:44 smithi012 ceph-mon[29863]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T04:59:45.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124"}]: dispatch 2024-03-26T04:59:45.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:45.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:44 smithi012 ceph-mon[41141]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T04:59:45.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124"}]: dispatch 2024-03-26T04:59:45.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:45.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:44 smithi186 ceph-mon[33438]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T04:59:45.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124"}]: dispatch 2024-03-26T04:59:46.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:46.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124"}]': finished 2024-03-26T04:59:46.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[29863]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T04:59:46.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[29863]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:59:46.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:46.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124"}]': finished 2024-03-26T04:59:46.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[41141]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T04:59:46.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:45 smithi012 ceph-mon[41141]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:59:46.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:45 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:46.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-124"}]': finished 2024-03-26T04:59:46.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:45 smithi186 ceph-mon[33438]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T04:59:46.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:45 smithi186 ceph-mon[33438]: pgmap v983: 204 pgs: 204 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:59:47.049 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:46 smithi012 ceph-mon[29863]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T04:59:47.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:46 smithi012 ceph-mon[41141]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T04:59:47.050 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:59:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:59:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:59:47.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:46 smithi186 ceph-mon[33438]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T04:59:48.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[29863]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[29863]: pgmap v986: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[41141]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[41141]: pgmap v986: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:48.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:47 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:48.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:47 smithi186 ceph-mon[33438]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T04:59:48.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:48.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:47 smithi186 ceph-mon[33438]: pgmap v986: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:48.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:47 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:48 smithi012 ceph-mon[29863]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T04:59:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:48 smithi012 ceph-mon[41141]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T04:59:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:49.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:49.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:48 smithi186 ceph-mon[33438]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T04:59:49.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:50.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-126"}]: dispatch 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: pgmap v989: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-126"}]': finished 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-126", "mode": "writeback"}]: dispatch 2024-03-26T04:59:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-126"}]: dispatch 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: pgmap v989: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-126"}]': finished 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T04:59:50.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-126", "mode": "writeback"}]: dispatch 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-126"}]: dispatch 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: pgmap v989: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-126"}]': finished 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T04:59:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-126", "mode": "writeback"}]: dispatch 2024-03-26T04:59:51.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-126", "mode": "writeback"}]': finished 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[29863]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-126", "mode": "writeback"}]': finished 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[41141]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T04:59:51.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:51.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:50 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:59:51.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-126", "mode": "writeback"}]': finished 2024-03-26T04:59:51.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:50 smithi186 ceph-mon[33438]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T04:59:51.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[29863]: pgmap v992: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[29863]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126"}]: dispatch 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[41141]: pgmap v992: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[41141]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T04:59:52.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126"}]: dispatch 2024-03-26T04:59:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:51 smithi186 ceph-mon[33438]: pgmap v992: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:59:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T04:59:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:51 smithi186 ceph-mon[33438]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T04:59:52.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126"}]: dispatch 2024-03-26T04:59:52.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 04:59:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T04:59:53.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126"}]': finished 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[29863]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126"}]': finished 2024-03-26T04:59:53.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:52 smithi012 ceph-mon[41141]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T04:59:53.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:52 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:53.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:52 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:59:53.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-126"}]': finished 2024-03-26T04:59:53.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:52 smithi186 ceph-mon[33438]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T04:59:53.761 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9048 ms) 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6036 ms) 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5049 ms) 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:59:53.762 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25148 ms) 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (12967 ms) 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12072 ms) 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 00000012 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12068 ms) 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:59:53.763 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6056 ms) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (556924 ms total) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1211 ms) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10852 ms) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18083 ms) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30149 ms total) 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-26T04:59:53.764 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7045 ms) 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8052 ms) 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: waiting for scrub... 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: done waiting 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24557 ms) 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9664 ms) 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7997 ms) 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8043 ms) 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-26T04:59:53.765 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10077 ms) 2024-03-26T04:59:53.766 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-26T04:59:54.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:54 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:54.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:54 smithi012 ceph-mon[29863]: pgmap v995: 204 pgs: 204 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:59:54.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:54 smithi012 ceph-mon[29863]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T04:59:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:54 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:54 smithi012 ceph-mon[41141]: pgmap v995: 204 pgs: 204 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:59:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:54 smithi012 ceph-mon[41141]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T04:59:54.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:54 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:59:54.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:54 smithi186 ceph-mon[33438]: pgmap v995: 204 pgs: 204 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:59:54.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:54 smithi186 ceph-mon[33438]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T04:59:56.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:55 smithi012 ceph-mon[29863]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T04:59:56.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:56.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:55 smithi012 ceph-mon[29863]: pgmap v998: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:56.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:55 smithi012 ceph-mon[41141]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T04:59:56.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:56.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:55 smithi012 ceph-mon[41141]: pgmap v998: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:56.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:55 smithi186 ceph-mon[33438]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T04:59:56.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:59:56.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:55 smithi186 ceph-mon[33438]: pgmap v998: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:59:57.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:04:59:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:59:57.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:57.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-mon[41141]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T04:59:57.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:57.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:57.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-mon[29863]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T04:59:57.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:57.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:59:57.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:56 smithi186 ceph-mon[33438]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T04:59:57.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:59:58.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-128"}]: dispatch 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: pgmap v1001: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T04:59:58.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-128"}]: dispatch 2024-03-26T04:59:58.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:58.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: pgmap v1001: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:59:58.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:58.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:58.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:58.150 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:57 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-128"}]: dispatch 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: pgmap v1001: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T04:59:58.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:57 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:59:59.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-128"}]': finished 2024-03-26T04:59:59.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:58 smithi012 ceph-mon[29863]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T04:59:59.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-128", "mode": "writeback"}]: dispatch 2024-03-26T04:59:59.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-128"}]': finished 2024-03-26T04:59:59.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:58 smithi012 ceph-mon[41141]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T04:59:59.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-128", "mode": "writeback"}]: dispatch 2024-03-26T04:59:59.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-128"}]': finished 2024-03-26T04:59:59.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:58 smithi186 ceph-mon[33438]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T04:59:59.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-128", "mode": "writeback"}]: dispatch 2024-03-26T05:00:00.108 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:59 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:00.108 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-128", "mode": "writeback"}]': finished 2024-03-26T05:00:00.108 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:59 smithi186 ceph-mon[33438]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T05:00:00.108 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 04:59:59 smithi186 ceph-mon[33438]: pgmap v1004: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:00.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:00.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-128", "mode": "writeback"}]': finished 2024-03-26T05:00:00.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[29863]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T05:00:00.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[29863]: pgmap v1004: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:00.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:00.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-128", "mode": "writeback"}]': finished 2024-03-26T05:00:00.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[41141]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T05:00:00.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 04:59:59 smithi012 ceph-mon[41141]: pgmap v1004: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:01.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 2 pool(s) do not have an application enabled 2024-03-26T05:00:01.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: pool 'test-rados-api-smithi012-78824-128' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: application not enabled on pool 'test-rados-api-smithi012-78824-111' 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[29863]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 2 pool(s) do not have an application enabled 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: pool 'test-rados-api-smithi012-78824-128' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: application not enabled on pool 'test-rados-api-smithi012-78824-111' 2024-03-26T05:00:01.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:00 smithi012 ceph-mon[41141]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T05:00:01.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 2 pool(s) do not have an application enabled 2024-03-26T05:00:01.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-26T05:00:01.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: pool 'test-rados-api-smithi012-78824-128' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-26T05:00:01.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-03-26T05:00:01.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T05:00:01.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: application not enabled on pool 'test-rados-api-smithi012-78824-111' 2024-03-26T05:00:01.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:00 smithi186 ceph-mon[33438]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T05:00:02.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:01 smithi012 ceph-mon[29863]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-26T05:00:02.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:01 smithi012 ceph-mon[41141]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-26T05:00:02.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:01 smithi186 ceph-mon[33438]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 654 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-03-26T05:00:02.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:02 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:02.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:00:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:00:03.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:02 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:03.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:02 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:04.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:04.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:03 smithi012 ceph-mon[29863]: pgmap v1006: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 328 B/s wr, 1 op/s 2024-03-26T05:00:04.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:04.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:03 smithi012 ceph-mon[41141]: pgmap v1006: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 328 B/s wr, 1 op/s 2024-03-26T05:00:04.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:04.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:03 smithi186 ceph-mon[33438]: pgmap v1006: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 328 B/s wr, 1 op/s 2024-03-26T05:00:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:04 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:05.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:05.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:04 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:05.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:04 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:05.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:06.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:06.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[29863]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T05:00:06.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128"}]: dispatch 2024-03-26T05:00:06.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[29863]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 283 B/s wr, 1 op/s 2024-03-26T05:00:06.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:06.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[41141]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T05:00:06.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128"}]: dispatch 2024-03-26T05:00:06.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:05 smithi012 ceph-mon[41141]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 283 B/s wr, 1 op/s 2024-03-26T05:00:06.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:06.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:05 smithi186 ceph-mon[33438]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T05:00:06.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128"}]: dispatch 2024-03-26T05:00:06.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:05 smithi186 ceph-mon[33438]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 283 B/s wr, 1 op/s 2024-03-26T05:00:07.093 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:00:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:00:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:00:07.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:07 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:07.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128"}]': finished 2024-03-26T05:00:07.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:07 smithi012 ceph-mon[41141]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T05:00:07.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:07 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:07.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128"}]': finished 2024-03-26T05:00:07.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:07 smithi012 ceph-mon[29863]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T05:00:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:07 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-128"}]': finished 2024-03-26T05:00:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:07 smithi186 ceph-mon[33438]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T05:00:08.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[41141]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T05:00:08.412 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[41141]: pgmap v1011: 172 pgs: 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:00:08.413 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[41141]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T05:00:08.413 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:08.413 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[29863]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T05:00:08.413 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[29863]: pgmap v1011: 172 pgs: 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:00:08.413 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[29863]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T05:00:08.413 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:08 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:08 smithi186 ceph-mon[33438]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T05:00:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:08 smithi186 ceph-mon[33438]: pgmap v1011: 172 pgs: 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:00:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:08 smithi186 ceph-mon[33438]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T05:00:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:08 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:10.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:10.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[29863]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T05:00:10.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:10.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[29863]: pgmap v1014: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:00:10.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:10.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[41141]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T05:00:10.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:10.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:09 smithi012 ceph-mon[41141]: pgmap v1014: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:00:10.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:10.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:09 smithi186 ceph-mon[33438]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T05:00:10.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:10.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:09 smithi186 ceph-mon[33438]: pgmap v1014: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:00:11.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:11.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:10 smithi012 ceph-mon[29863]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T05:00:11.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-130"}]: dispatch 2024-03-26T05:00:11.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:11.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:10 smithi012 ceph-mon[41141]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T05:00:11.151 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-130"}]: dispatch 2024-03-26T05:00:11.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:11.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:10 smithi186 ceph-mon[33438]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T05:00:11.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-130"}]: dispatch 2024-03-26T05:00:12.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-130"}]': finished 2024-03-26T05:00:12.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[29863]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T05:00:12.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-130", "mode": "writeback"}]: dispatch 2024-03-26T05:00:12.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[29863]: pgmap v1017: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:12.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-130"}]': finished 2024-03-26T05:00:12.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[41141]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T05:00:12.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-130", "mode": "writeback"}]: dispatch 2024-03-26T05:00:12.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:11 smithi012 ceph-mon[41141]: pgmap v1017: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:12.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-130"}]': finished 2024-03-26T05:00:12.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:11 smithi186 ceph-mon[33438]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T05:00:12.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-130", "mode": "writeback"}]: dispatch 2024-03-26T05:00:12.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:11 smithi186 ceph-mon[33438]: pgmap v1017: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 676 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:12.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:00:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:00:12.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:12 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:12.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-130", "mode": "writeback"}]': finished 2024-03-26T05:00:12.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:12 smithi186 ceph-mon[33438]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T05:00:12.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:13.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:13.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-130", "mode": "writeback"}]': finished 2024-03-26T05:00:13.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[29863]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T05:00:13.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:13.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:13.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-130", "mode": "writeback"}]': finished 2024-03-26T05:00:13.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[41141]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T05:00:13.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:14.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:14.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:14.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130"}]: dispatch 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: pgmap v1020: 204 pgs: 204 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130"}]': finished 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130"}]: dispatch 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: pgmap v1020: 204 pgs: 204 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:14.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130"}]': finished 2024-03-26T05:00:14.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T05:00:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130"}]: dispatch 2024-03-26T05:00:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: pgmap v1020: 204 pgs: 204 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-130"}]': finished 2024-03-26T05:00:15.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:14 smithi012 ceph-mon[29863]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T05:00:15.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:14 smithi012 ceph-mon[41141]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T05:00:15.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:14 smithi186 ceph-mon[33438]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T05:00:16.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:15 smithi012 ceph-mon[29863]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T05:00:16.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:15 smithi012 ceph-mon[29863]: pgmap v1023: 172 pgs: 172 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:16.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:15 smithi012 ceph-mon[41141]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T05:00:16.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:15 smithi012 ceph-mon[41141]: pgmap v1023: 172 pgs: 172 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:16.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:15 smithi186 ceph-mon[33438]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T05:00:16.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:15 smithi186 ceph-mon[33438]: pgmap v1023: 172 pgs: 172 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:17.049 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:16 smithi012 ceph-mon[29863]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T05:00:17.050 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:17.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:16 smithi012 ceph-mon[41141]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T05:00:17.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:17.050 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:00:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:00:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:00:17.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:16 smithi186 ceph-mon[33438]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T05:00:17.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: pgmap v1026: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T05:00:18.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T05:00:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: pgmap v1026: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: pgmap v1026: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T05:00:18.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:20.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:20.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[29863]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-132", "mode": "writeback"}]: dispatch 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[29863]: pgmap v1029: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[41141]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-132", "mode": "writeback"}]: dispatch 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:20.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:19 smithi012 ceph-mon[41141]: pgmap v1029: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:00:20.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:20.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:19 smithi186 ceph-mon[33438]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T05:00:20.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-132", "mode": "writeback"}]: dispatch 2024-03-26T05:00:20.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:20.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:20.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:19 smithi186 ceph-mon[33438]: pgmap v1029: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T05:00:21.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:20 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:21.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-132", "mode": "writeback"}]': finished 2024-03-26T05:00:21.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:20 smithi186 ceph-mon[33438]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T05:00:21.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:20 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:21.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-132", "mode": "writeback"}]': finished 2024-03-26T05:00:21.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:20 smithi012 ceph-mon[29863]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T05:00:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:20 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-132", "mode": "writeback"}]': finished 2024-03-26T05:00:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:20 smithi012 ceph-mon[41141]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T05:00:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:21 smithi186 ceph-mon[33438]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T05:00:22.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:21 smithi186 ceph-mon[33438]: pgmap v1032: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:22.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:21 smithi012 ceph-mon[29863]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T05:00:22.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:21 smithi012 ceph-mon[29863]: pgmap v1032: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:22.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:21 smithi012 ceph-mon[41141]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T05:00:22.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:21 smithi012 ceph-mon[41141]: pgmap v1032: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:22.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:00:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:00:23.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:22 smithi186 ceph-mon[33438]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T05:00:23.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:22 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:23.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:22 smithi012 ceph-mon[29863]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T05:00:23.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:22 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:23.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:22 smithi012 ceph-mon[41141]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T05:00:23.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:22 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:23 smithi186 ceph-mon[33438]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T05:00:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:24.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:23 smithi186 ceph-mon[33438]: pgmap v1035: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:00:24.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:24.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:24.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[29863]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T05:00:24.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:24.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[29863]: pgmap v1035: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:00:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[41141]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T05:00:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:24.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:23 smithi012 ceph-mon[41141]: pgmap v1035: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:00:25.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:25.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:24 smithi186 ceph-mon[33438]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T05:00:25.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:25.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:24 smithi012 ceph-mon[29863]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T05:00:25.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:25.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:24 smithi012 ceph-mon[41141]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T05:00:26.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:25 smithi186 ceph-mon[33438]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T05:00:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:26.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:25 smithi186 ceph-mon[33438]: pgmap v1038: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:00:26.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:25 smithi012 ceph-mon[29863]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T05:00:26.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:26.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:25 smithi012 ceph-mon[29863]: pgmap v1038: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:00:26.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:25 smithi012 ceph-mon[41141]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T05:00:26.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:26.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:25 smithi012 ceph-mon[41141]: pgmap v1038: 204 pgs: 204 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:00:26.966 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:00:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:00:27.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:27.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-mon[29863]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T05:00:27.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:27.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:27.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-mon[41141]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T05:00:27.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:27.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:27.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:26 smithi186 ceph-mon[33438]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T05:00:27.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132"}]: dispatch 2024-03-26T05:00:28.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:28.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:28.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[29863]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T05:00:28.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[29863]: pgmap v1041: 204 pgs: 204 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:00:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[41141]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T05:00:28.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:27 smithi012 ceph-mon[41141]: pgmap v1041: 204 pgs: 204 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:00:28.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:27 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:28.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-132"}]': finished 2024-03-26T05:00:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:27 smithi186 ceph-mon[33438]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T05:00:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:27 smithi186 ceph-mon[33438]: pgmap v1041: 204 pgs: 204 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T05:00:29.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:28 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:29.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:28 smithi012 ceph-mon[29863]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T05:00:29.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:28 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:29.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:28 smithi012 ceph-mon[41141]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T05:00:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:28 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:29.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:28 smithi186 ceph-mon[33438]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T05:00:30.358 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:29 smithi186 ceph-mon[33438]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T05:00:30.358 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:30.358 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:29 smithi186 ceph-mon[33438]: pgmap v1044: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:00:30.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:29 smithi012 ceph-mon[29863]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T05:00:30.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:30.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:29 smithi012 ceph-mon[29863]: pgmap v1044: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:00:30.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:29 smithi012 ceph-mon[41141]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T05:00:30.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:30.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:29 smithi012 ceph-mon[41141]: pgmap v1044: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:00:31.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:31.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:31 smithi012 ceph-mon[29863]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T05:00:31.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:31.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:31.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:31 smithi012 ceph-mon[41141]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T05:00:31.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:31.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:31.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:31 smithi186 ceph-mon[33438]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T05:00:31.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:32.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:32.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[29863]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T05:00:32.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-134"}]: dispatch 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[29863]: pgmap v1047: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-134"}]': finished 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[29863]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[41141]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-134"}]: dispatch 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[41141]: pgmap v1047: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-134"}]': finished 2024-03-26T05:00:32.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:32 smithi012 ceph-mon[41141]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T05:00:32.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:32.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-mon[33438]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T05:00:32.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-134"}]: dispatch 2024-03-26T05:00:32.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-mon[33438]: pgmap v1047: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:32.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-134"}]': finished 2024-03-26T05:00:32.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-mon[33438]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T05:00:32.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:00:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:00:33.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-134", "mode": "writeback"}]: dispatch 2024-03-26T05:00:33.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:33.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-134", "mode": "writeback"}]': finished 2024-03-26T05:00:33.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[29863]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T05:00:33.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:33.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-134", "mode": "writeback"}]: dispatch 2024-03-26T05:00:33.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:33.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-134", "mode": "writeback"}]': finished 2024-03-26T05:00:33.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[41141]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T05:00:33.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:33 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:33.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-134", "mode": "writeback"}]: dispatch 2024-03-26T05:00:33.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:33 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:33.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-134", "mode": "writeback"}]': finished 2024-03-26T05:00:33.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:33 smithi186 ceph-mon[33438]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T05:00:33.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:33 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:34.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:34.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[29863]: pgmap v1050: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:34.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[29863]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134"}]: dispatch 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[41141]: pgmap v1050: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[41141]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T05:00:34.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:34 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134"}]: dispatch 2024-03-26T05:00:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:34 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:34 smithi186 ceph-mon[33438]: pgmap v1050: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:34 smithi186 ceph-mon[33438]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T05:00:34.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:34 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134"}]: dispatch 2024-03-26T05:00:35.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:35.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:35.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:35.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134"}]': finished 2024-03-26T05:00:35.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[29863]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T05:00:35.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:35.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:35.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:35.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134"}]': finished 2024-03-26T05:00:35.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:35 smithi012 ceph-mon[41141]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T05:00:35.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:35 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:35.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:35 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:35.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:35 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:35.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-134"}]': finished 2024-03-26T05:00:35.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:35 smithi186 ceph-mon[33438]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T05:00:36.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-mon[41141]: pgmap v1053: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:36.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:36.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-mon[41141]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T05:00:36.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-mon[29863]: pgmap v1053: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:36.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:36.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-mon[29863]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T05:00:36.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:36 smithi186 ceph-mon[33438]: pgmap v1053: 204 pgs: 204 active+clean; 582 KiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:36.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:36 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:36.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:36 smithi186 ceph-mon[33438]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T05:00:37.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:00:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:00:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:00:38.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:37 smithi012 ceph-mon[29863]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T05:00:38.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:38.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:37 smithi012 ceph-mon[29863]: pgmap v1056: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:00:38.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:37 smithi012 ceph-mon[41141]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T05:00:38.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:38.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:37 smithi012 ceph-mon[41141]: pgmap v1056: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:00:38.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:37 smithi186 ceph-mon[33438]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T05:00:38.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:38.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:37 smithi186 ceph-mon[33438]: pgmap v1056: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:00:39.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:39.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:38 smithi012 ceph-mon[29863]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T05:00:39.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:39.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:39.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:38 smithi012 ceph-mon[41141]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T05:00:39.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:39.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:39.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:38 smithi186 ceph-mon[33438]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T05:00:39.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:40.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:40.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[29863]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T05:00:40.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-136"}]: dispatch 2024-03-26T05:00:40.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[29863]: pgmap v1059: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:00:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[41141]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T05:00:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-136"}]: dispatch 2024-03-26T05:00:40.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:39 smithi012 ceph-mon[41141]: pgmap v1059: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:00:40.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:40.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:39 smithi186 ceph-mon[33438]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T05:00:40.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-136"}]: dispatch 2024-03-26T05:00:40.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:39 smithi186 ceph-mon[33438]: pgmap v1059: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T05:00:41.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-136"}]': finished 2024-03-26T05:00:41.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:40 smithi012 ceph-mon[29863]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T05:00:41.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-136", "mode": "writeback"}]: dispatch 2024-03-26T05:00:41.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-136"}]': finished 2024-03-26T05:00:41.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:40 smithi012 ceph-mon[41141]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T05:00:41.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-136", "mode": "writeback"}]: dispatch 2024-03-26T05:00:41.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-136"}]': finished 2024-03-26T05:00:41.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:40 smithi186 ceph-mon[33438]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T05:00:41.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-136", "mode": "writeback"}]: dispatch 2024-03-26T05:00:42.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-136", "mode": "writeback"}]': finished 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[29863]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[29863]: pgmap v1062: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-136", "mode": "writeback"}]': finished 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[41141]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:42.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:41 smithi012 ceph-mon[41141]: pgmap v1062: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:41 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:00:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-136", "mode": "writeback"}]': finished 2024-03-26T05:00:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:41 smithi186 ceph-mon[33438]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T05:00:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:42.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:41 smithi186 ceph-mon[33438]: pgmap v1062: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 706 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:42.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:42.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:42 smithi186 ceph-mon[33438]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T05:00:42.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136"}]: dispatch 2024-03-26T05:00:42.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:42 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:42.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:00:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:00:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[29863]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T05:00:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136"}]: dispatch 2024-03-26T05:00:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:43.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:00:43.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[41141]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T05:00:43.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136"}]: dispatch 2024-03-26T05:00:43.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:42 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136"}]': finished 2024-03-26T05:00:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[29863]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T05:00:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[29863]: pgmap v1065: 204 pgs: 204 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T05:00:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136"}]': finished 2024-03-26T05:00:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[41141]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T05:00:44.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:43 smithi012 ceph-mon[41141]: pgmap v1065: 204 pgs: 204 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T05:00:44.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:43 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:00:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-136"}]': finished 2024-03-26T05:00:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:43 smithi186 ceph-mon[33438]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T05:00:44.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:43 smithi186 ceph-mon[33438]: pgmap v1065: 204 pgs: 204 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T05:00:45.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:44 smithi012 ceph-mon[29863]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T05:00:45.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:44 smithi012 ceph-mon[41141]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T05:00:45.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:44 smithi186 ceph-mon[33438]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T05:00:46.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:45 smithi012 ceph-mon[29863]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T05:00:46.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:46.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:45 smithi012 ceph-mon[29863]: pgmap v1068: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T05:00:46.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:45 smithi012 ceph-mon[41141]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T05:00:46.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:46.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:45 smithi012 ceph-mon[41141]: pgmap v1068: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T05:00:46.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:45 smithi186 ceph-mon[33438]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T05:00:46.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:46.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:45 smithi186 ceph-mon[33438]: pgmap v1068: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-26T05:00:47.049 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:47.050 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-mon[29863]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T05:00:47.050 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:47.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:47.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-mon[41141]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T05:00:47.050 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:47.050 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:00:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:00:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:00:47.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:47.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:46 smithi186 ceph-mon[33438]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T05:00:47.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:48.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:48.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[41141]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T05:00:48.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:00:48.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[41141]: pgmap v1071: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:48.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[29863]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T05:00:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:00:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[29863]: pgmap v1071: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:48.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:47 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:48.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:48.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:47 smithi186 ceph-mon[33438]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T05:00:48.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:00:48.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:47 smithi186 ceph-mon[33438]: pgmap v1071: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:48.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:47 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:00:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[29863]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T05:00:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:00:49.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:00:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:00:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[41141]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T05:00:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:00:49.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:00:49.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:00:49.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:48 smithi186 ceph-mon[33438]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T05:00:49.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:00:49.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:00:50.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T05:00:50.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:00:50.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: pgmap v1074: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:50.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:50.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[29863]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: pgmap v1074: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:00:50.149 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:49 smithi012 ceph-mon[41141]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T05:00:50.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T05:00:50.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T05:00:50.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: pgmap v1074: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:00:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T05:00:50.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:49 smithi186 ceph-mon[33438]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T05:00:51.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:51.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138"}]: dispatch 2024-03-26T05:00:51.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:51.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138"}]: dispatch 2024-03-26T05:00:51.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:00:51.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138"}]: dispatch 2024-03-26T05:00:52.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138"}]': finished 2024-03-26T05:00:52.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:51 smithi186 ceph-mon[33438]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T05:00:52.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:51 smithi186 ceph-mon[33438]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:52.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138"}]': finished 2024-03-26T05:00:52.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:51 smithi012 ceph-mon[41141]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T05:00:52.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:51 smithi012 ceph-mon[41141]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:52.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-138"}]': finished 2024-03-26T05:00:52.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:51 smithi012 ceph-mon[29863]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T05:00:52.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:51 smithi012 ceph-mon[29863]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:00:52.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:00:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:00:53.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:52 smithi186 ceph-mon[33438]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T05:00:53.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:52 smithi012 ceph-mon[29863]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T05:00:53.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:52 smithi012 ceph-mon[41141]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T05:00:54.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:54.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[29863]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T05:00:54.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:54.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[29863]: pgmap v1080: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[41141]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T05:00:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:54.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:53 smithi012 ceph-mon[41141]: pgmap v1080: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:54.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:00:54.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:53 smithi186 ceph-mon[33438]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T05:00:54.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:00:54.613 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:53 smithi186 ceph-mon[33438]: pgmap v1080: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:55.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:55.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:54 smithi012 ceph-mon[29863]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T05:00:55.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:54 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:55.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:55.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:54 smithi012 ceph-mon[41141]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T05:00:55.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:54 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:55.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:00:55.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:54 smithi186 ceph-mon[33438]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T05:00:55.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:54 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:00:56.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:56.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[29863]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T05:00:56.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:00:56.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[29863]: pgmap v1083: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:56.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:56.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[41141]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T05:00:56.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:00:56.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:55 smithi012 ceph-mon[41141]: pgmap v1083: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:00:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:55 smithi186 ceph-mon[33438]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T05:00:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T05:00:56.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:55 smithi186 ceph-mon[33438]: pgmap v1083: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:56.989 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:00:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:00:57.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:57.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:00:57.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[29863]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T05:00:57.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:00:57.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:57.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:00:57.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[41141]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T05:00:57.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:56 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:00:57.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:56 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:00:57.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T05:00:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:56 smithi186 ceph-mon[33438]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T05:00:57.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:56 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:00:58.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T05:00:58.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:58 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:00:58.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T05:00:58.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T05:00:58.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:00:58.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:58.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:00:58.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:00:58.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T05:00:58.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:58 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T05:00:59.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:00:59 smithi186 ceph-mon[33438]: pgmap v1089: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:59.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[41141]: pgmap v1089: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T05:00:59.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:00:59 smithi012 ceph-mon[29863]: pgmap v1089: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:02.160 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:02 smithi186 ceph-mon[33438]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-26T05:01:02.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:02 smithi012 ceph-mon[29863]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-26T05:01:02.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:02 smithi012 ceph-mon[41141]: pgmap v1090: 204 pgs: 204 active+clean; 4.4 MiB data, 722 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-26T05:01:02.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:01:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:01:03.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:03.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:03 smithi012 ceph-mon[29863]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T05:01:03.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:03.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:03 smithi012 ceph-mon[41141]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T05:01:03.437 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:03.437 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:03 smithi186 ceph-mon[33438]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T05:01:04.380 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:04.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:04.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:05.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:05 smithi012 ceph-mon[29863]: pgmap v1092: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-26T05:01:05.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:05 smithi012 ceph-mon[41141]: pgmap v1092: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-26T05:01:05.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:05 smithi186 ceph-mon[33438]: pgmap v1092: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-26T05:01:07.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:01:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:01:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:08.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:08 smithi012 ceph-mon[29863]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.3 KiB/s wr, 2 op/s 2024-03-26T05:01:08.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:08 smithi012 ceph-mon[41141]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.3 KiB/s wr, 2 op/s 2024-03-26T05:01:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:08 smithi186 ceph-mon[33438]: pgmap v1093: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.3 KiB/s wr, 2 op/s 2024-03-26T05:01:09.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:09 smithi012 ceph-mon[29863]: pgmap v1094: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-03-26T05:01:09.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:09 smithi012 ceph-mon[41141]: pgmap v1094: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-03-26T05:01:09.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:09 smithi186 ceph-mon[33438]: pgmap v1094: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 6.6 KiB/s wr, 1 op/s 2024-03-26T05:01:11.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:11.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140"}]: dispatch 2024-03-26T05:01:11.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:11.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140"}]: dispatch 2024-03-26T05:01:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140"}]: dispatch 2024-03-26T05:01:12.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:12 smithi012 ceph-mon[29863]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-03-26T05:01:12.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140"}]': finished 2024-03-26T05:01:12.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:12 smithi012 ceph-mon[29863]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T05:01:12.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:12 smithi012 ceph-mon[41141]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-03-26T05:01:12.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140"}]': finished 2024-03-26T05:01:12.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:12 smithi012 ceph-mon[41141]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T05:01:12.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:12 smithi186 ceph-mon[33438]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-03-26T05:01:12.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-140"}]': finished 2024-03-26T05:01:12.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:12 smithi186 ceph-mon[33438]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T05:01:12.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:01:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:01:13.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:13 smithi012 ceph-mon[29863]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T05:01:13.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:13.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:13 smithi012 ceph-mon[29863]: pgmap v1098: 172 pgs: 172 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:13.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:13 smithi012 ceph-mon[41141]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T05:01:13.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:13.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:13 smithi012 ceph-mon[41141]: pgmap v1098: 172 pgs: 172 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:13 smithi186 ceph-mon[33438]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T05:01:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:13 smithi186 ceph-mon[33438]: pgmap v1098: 172 pgs: 172 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:14.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:14 smithi186 ceph-mon[33438]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T05:01:14.207 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:14 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:14.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:14 smithi012 ceph-mon[29863]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T05:01:14.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:14 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:14.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:14 smithi012 ceph-mon[41141]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T05:01:14.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:14 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:15.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:15.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[29863]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T05:01:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:15.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[29863]: pgmap v1101: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[41141]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T05:01:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:15.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:15 smithi012 ceph-mon[41141]: pgmap v1101: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:15 smithi186 ceph-mon[33438]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T05:01:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:15 smithi186 ceph-mon[33438]: pgmap v1101: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:16.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:16.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-mon[29863]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T05:01:16.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-mon[41141]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T05:01:16.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:16.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:16.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:16 smithi186 ceph-mon[33438]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T05:01:16.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:16 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:17.129 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:01:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:01:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:17.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-142"}]': finished 2024-03-26T05:01:17.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[29863]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T05:01:17.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-142", "mode": "writeback"}]: dispatch 2024-03-26T05:01:17.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[29863]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:01:17.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:17.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-142", "mode": "writeback"}]': finished 2024-03-26T05:01:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-142"}]': finished 2024-03-26T05:01:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[41141]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T05:01:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-142", "mode": "writeback"}]: dispatch 2024-03-26T05:01:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[41141]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:01:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:17.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-142", "mode": "writeback"}]': finished 2024-03-26T05:01:17.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-142"}]': finished 2024-03-26T05:01:17.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:17 smithi186 ceph-mon[33438]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T05:01:17.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-142", "mode": "writeback"}]: dispatch 2024-03-26T05:01:17.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:17 smithi186 ceph-mon[33438]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:01:17.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:17 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:17.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-142", "mode": "writeback"}]': finished 2024-03-26T05:01:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:18 smithi012 ceph-mon[29863]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T05:01:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:01:18.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:01:18.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:18 smithi012 ceph-mon[41141]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T05:01:18.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:01:18.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:01:18.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:18 smithi186 ceph-mon[33438]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T05:01:18.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:01:18.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:01:19.395 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T05:01:19.396 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:01:19.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T05:01:19.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:01:19.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T05:01:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:01:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:01:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T05:01:19.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:01:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:01:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[29863]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T05:01:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:01:20.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:01:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[41141]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T05:01:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:01:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:20 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:01:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:20 smithi186 ceph-mon[33438]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T05:01:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:01:21.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[29863]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:21.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:01:21.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[29863]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T05:01:21.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:01:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[41141]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:01:21.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[41141]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T05:01:21.400 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:01:21.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:21 smithi186 ceph-mon[33438]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail 2024-03-26T05:01:21.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:01:21.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:21 smithi186 ceph-mon[33438]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T05:01:21.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T05:01:22.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:01:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:01:23.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:01:23.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[41141]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T05:01:23.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:01:23.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:23.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[41141]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:23.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:01:23.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[29863]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T05:01:23.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:01:23.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:23.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:23 smithi012 ceph-mon[29863]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:23.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T05:01:23.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:23 smithi186 ceph-mon[33438]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T05:01:23.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T05:01:23.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:23.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:23 smithi186 ceph-mon[33438]: pgmap v1113: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:24.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:01:24.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:24 smithi012 ceph-mon[41141]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T05:01:24.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:24.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:01:24.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:24 smithi012 ceph-mon[29863]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T05:01:24.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:24.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T05:01:24.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:24 smithi186 ceph-mon[33438]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T05:01:24.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:25.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:01:25.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:25 smithi186 ceph-mon[33438]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T05:01:25.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:25.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:25 smithi186 ceph-mon[33438]: pgmap v1116: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:25.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]': finished 2024-03-26T05:01:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:01:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[29863]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T05:01:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[29863]: pgmap v1116: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]': finished 2024-03-26T05:01:25.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:01:25.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[41141]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T05:01:25.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:25.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[41141]: pgmap v1116: 204 pgs: 204 active+clean; 4.4 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:01:25.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]': finished 2024-03-26T05:01:26.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:26 smithi186 ceph-mon[33438]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T05:01:26.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:26.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-mon[29863]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T05:01:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-mon[41141]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T05:01:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-142"}]: dispatch 2024-03-26T05:01:27.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:01:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:01:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:27.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:27 smithi186 ceph-mon[33438]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T05:01:27.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:27 smithi186 ceph-mon[33438]: pgmap v1119: 172 pgs: 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:01:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:27 smithi012 ceph-mon[29863]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T05:01:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:27 smithi012 ceph-mon[29863]: pgmap v1119: 172 pgs: 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:01:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:27 smithi012 ceph-mon[41141]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T05:01:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:27 smithi012 ceph-mon[41141]: pgmap v1119: 172 pgs: 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:01:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:28 smithi186 ceph-mon[33438]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T05:01:28.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:28.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:28 smithi012 ceph-mon[29863]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T05:01:28.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:28.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:28 smithi012 ceph-mon[41141]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T05:01:28.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[41141]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[41141]: pgmap v1122: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[29863]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:29.455 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:29 smithi012 ceph-mon[29863]: pgmap v1122: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:01:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:29 smithi186 ceph-mon[33438]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T05:01:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:29 smithi186 ceph-mon[33438]: pgmap v1122: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T05:01:30.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:30.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:30 smithi186 ceph-mon[33438]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T05:01:30.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:30 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:30.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:30.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:30 smithi012 ceph-mon[29863]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T05:01:30.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:30 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:30.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:30.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:30 smithi012 ceph-mon[41141]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T05:01:30.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:30 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:31.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-144"}]': finished 2024-03-26T05:01:31.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[29863]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T05:01:31.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:01:31.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[29863]: pgmap v1125: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-144", "mode": "readproxy"}]': finished 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-144"}]': finished 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[41141]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[41141]: pgmap v1125: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:31.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:31 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-144", "mode": "readproxy"}]': finished 2024-03-26T05:01:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-144"}]': finished 2024-03-26T05:01:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:31 smithi186 ceph-mon[33438]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T05:01:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-144", "mode": "readproxy"}]: dispatch 2024-03-26T05:01:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:31 smithi186 ceph-mon[33438]: pgmap v1125: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 743 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:01:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:31 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:31 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-144", "mode": "readproxy"}]': finished 2024-03-26T05:01:32.607 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:32 smithi186 ceph-mon[33438]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T05:01:32.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:32 smithi012 ceph-mon[29863]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T05:01:32.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:32 smithi012 ceph-mon[41141]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T05:01:32.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:01:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:01:33.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:33.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:33 smithi012 ceph-mon[29863]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s 2024-03-26T05:01:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:33 smithi012 ceph-mon[41141]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s 2024-03-26T05:01:33.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:33.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:33 smithi186 ceph-mon[33438]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s 2024-03-26T05:01:34.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:34.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:34.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:35.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:35 smithi012 ceph-mon[29863]: pgmap v1128: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T05:01:35.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:35 smithi012 ceph-mon[41141]: pgmap v1128: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T05:01:35.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:35 smithi186 ceph-mon[33438]: pgmap v1128: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T05:01:37.145 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:01:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:01:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:37.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:37 smithi012 ceph-mon[29863]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-03-26T05:01:37.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:37 smithi012 ceph-mon[41141]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-03-26T05:01:37.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:37 smithi186 ceph-mon[33438]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-03-26T05:01:39.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:39 smithi012 ceph-mon[29863]: pgmap v1130: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-03-26T05:01:39.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:39 smithi012 ceph-mon[41141]: pgmap v1130: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-03-26T05:01:39.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:39 smithi186 ceph-mon[33438]: pgmap v1130: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-03-26T05:01:41.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:41 smithi012 ceph-mon[29863]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-26T05:01:41.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:41 smithi012 ceph-mon[41141]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-26T05:01:41.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:41 smithi186 ceph-mon[33438]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-26T05:01:42.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:42.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:42.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:43.151 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:01:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:01:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:01:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:43 smithi186 ceph-mon[33438]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T05:01:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:43 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:43 smithi186 ceph-mon[33438]: pgmap v1133: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:43.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:01:43.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[29863]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T05:01:43.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:43.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:43.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[29863]: pgmap v1133: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:43.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:01:43.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[41141]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T05:01:43.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:43.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:43.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:43 smithi012 ceph-mon[41141]: pgmap v1133: 204 pgs: 204 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T05:01:44.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:44 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:44.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]': finished 2024-03-26T05:01:44.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:44 smithi186 ceph-mon[33438]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T05:01:44.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:44.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]': finished 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[29863]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:44.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]': finished 2024-03-26T05:01:44.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[41141]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T05:01:44.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:01:44.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-144"}]: dispatch 2024-03-26T05:01:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:45 smithi186 ceph-mon[33438]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T05:01:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:45 smithi186 ceph-mon[33438]: pgmap v1136: 172 pgs: 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:45.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:45 smithi012 ceph-mon[29863]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T05:01:45.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:45 smithi012 ceph-mon[29863]: pgmap v1136: 172 pgs: 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:45.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:45 smithi012 ceph-mon[41141]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T05:01:45.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:45 smithi012 ceph-mon[41141]: pgmap v1136: 172 pgs: 172 active+clean; 4.4 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:01:46.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:46 smithi186 ceph-mon[33438]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T05:01:46.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:46.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:46 smithi012 ceph-mon[29863]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T05:01:46.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:46.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:46 smithi012 ceph-mon[41141]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T05:01:46.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:01:47.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:01:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:01:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:47.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:47.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:47 smithi186 ceph-mon[33438]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T05:01:47.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:47.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:47 smithi186 ceph-mon[33438]: pgmap v1139: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[29863]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[29863]: pgmap v1139: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[41141]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:01:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:47 smithi012 ceph-mon[41141]: pgmap v1139: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:01:48.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:48.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:48 smithi186 ceph-mon[33438]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T05:01:48.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:48 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:01:48.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:48.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:48 smithi012 ceph-mon[29863]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T05:01:48.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:48 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:01:48.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:01:48.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:48 smithi012 ceph-mon[41141]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T05:01:48.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:48 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:01:49.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-146"}]': finished 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-146", "mode": "writeback"}]: dispatch 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: pgmap v1142: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:49.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:49 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-146", "mode": "writeback"}]': finished 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-146"}]': finished 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-146", "mode": "writeback"}]: dispatch 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: pgmap v1142: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-146", "mode": "writeback"}]': finished 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi012-78824-111", "overlaypool": "test-rados-api-smithi012-78824-146"}]': finished 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-146", "mode": "writeback"}]: dispatch 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: pgmap v1142: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T05:01:49.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:49 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi012-78824-146", "mode": "writeback"}]': finished 2024-03-26T05:01:50.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:50 smithi186 ceph-mon[33438]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T05:01:50.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:50 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:01:50.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:50 smithi012 ceph-mon[29863]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T05:01:50.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:50 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:01:50.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:50 smithi012 ceph-mon[41141]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T05:01:50.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:50 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T05:01:51.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:01:51.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[29863]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T05:01:51.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:01:51.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[29863]: pgmap v1145: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:01:51.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:01:51.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[41141]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T05:01:51.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:01:51.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:51 smithi012 ceph-mon[41141]: pgmap v1145: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:01:51.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T05:01:51.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:51 smithi186 ceph-mon[33438]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T05:01:51.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:51 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T05:01:51.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:51 smithi186 ceph-mon[33438]: pgmap v1145: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:01:52.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:01:52.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:52 smithi012 ceph-mon[41141]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T05:01:52.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:52 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:01:52.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:01:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:52 smithi012 ceph-mon[29863]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T05:01:52.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:52 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:01:52.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T05:01:52.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:52 smithi186 ceph-mon[33438]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T05:01:52.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:52 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T05:01:53.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:01:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: pgmap v1148: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:01:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: pgmap v1148: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T05:01:53.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:53 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:01:53.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T05:01:53.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T05:01:53.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T05:01:53.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T05:01:53.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:01:53.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: pgmap v1148: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:01:53.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T05:01:53.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T05:01:53.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:53 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T05:01:55.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:55 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:01:55.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:55 smithi012 ceph-mon[29863]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T05:01:55.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:55 smithi012 ceph-mon[29863]: pgmap v1151: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:01:55.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:55 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:01:55.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:55 smithi012 ceph-mon[41141]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T05:01:55.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:55 smithi012 ceph-mon[41141]: pgmap v1151: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:01:55.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:55 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T05:01:55.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:55 smithi186 ceph-mon[33438]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T05:01:55.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:55 smithi186 ceph-mon[33438]: pgmap v1151: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T05:01:56.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:56 smithi012 ceph-mon[29863]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:01:56.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:56 smithi012 ceph-mon[41141]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:01:56.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:56 smithi186 ceph-mon[33438]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T05:01:57.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:01:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:01:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:01:57.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:57 smithi012 ceph-mon[29863]: pgmap v1152: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 878 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:01:57.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:57 smithi012 ceph-mon[41141]: pgmap v1152: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 878 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:01:57.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:57 smithi186 ceph-mon[33438]: pgmap v1152: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 878 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:01:58.706 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:01:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=sqlstore.transactions t=2024-03-26T05:01:58.382173666Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T05:01:59.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:58 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:59.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:58 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:01:59.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:58 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:02:00.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:59 smithi012 ceph-mon[29863]: pgmap v1153: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:00.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:59 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:00.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:01:59 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:00.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:59 smithi012 ceph-mon[41141]: pgmap v1153: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:00.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:59 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:00.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:01:59 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:00.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:59 smithi186 ceph-mon[33438]: pgmap v1153: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:00.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:59 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:00.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:01:59 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:01.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:02:01.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:02:01.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:00 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:01.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:02:01.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:02:01.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:00 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:02:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:02:01.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:00 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:02.357 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:02 smithi186 ceph-mon[33438]: pgmap v1154: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T05:02:02.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:02 smithi012 ceph-mon[29863]: pgmap v1154: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T05:02:02.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:02 smithi012 ceph-mon[41141]: pgmap v1154: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T05:02:03.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:02:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:02:03.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:03.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:03 smithi012 ceph-mon[41141]: pgmap v1155: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:03.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:03.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:03 smithi012 ceph-mon[29863]: pgmap v1155: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:03 smithi186 ceph-mon[33438]: pgmap v1155: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:05.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:05.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:05.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:05.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[41141]: pgmap v1156: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:05.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:05.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:05.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:05.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:05 smithi012 ceph-mon[29863]: pgmap v1156: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:05.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:05 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:05.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:05 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:05.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:05 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:05.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:05 smithi186 ceph-mon[33438]: pgmap v1156: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 945 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T05:02:06.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:06.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-mon[29863]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T05:02:06.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:02:06.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:06.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-mon[41141]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T05:02:06.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:02:06.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:06.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:06 smithi186 ceph-mon[33438]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T05:02:06.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:06 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:02:07.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:02:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:02:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]': finished 2024-03-26T05:02:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:07 smithi186 ceph-mon[33438]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T05:02:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:07 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:02:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:07 smithi186 ceph-mon[33438]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]': finished 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[29863]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[29863]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]': finished 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[41141]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T05:02:07.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:07.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-146"}]: dispatch 2024-03-26T05:02:07.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:07 smithi012 ceph-mon[41141]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T05:02:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:08 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:08 smithi186 ceph-mon[33438]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:02:08.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:08 smithi186 ceph-mon[33438]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T05:02:08.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:08 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:08.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:08 smithi012 ceph-mon[29863]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:02:08.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:08 smithi012 ceph-mon[29863]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T05:02:08.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:08 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:08.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:08 smithi012 ceph-mon[41141]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T05:02:08.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:08 smithi012 ceph-mon[41141]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T05:02:09.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:09 smithi186 ceph-mon[33438]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T05:02:09.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:09 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:09.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:09 smithi186 ceph-mon[33438]: pgmap v1162: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:02:09.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:09 smithi012 ceph-mon[41141]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T05:02:09.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:09 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:09.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:09 smithi012 ceph-mon[41141]: pgmap v1162: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:02:09.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:09 smithi012 ceph-mon[29863]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T05:02:09.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:09 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:09.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:09 smithi012 ceph-mon[29863]: pgmap v1162: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T05:02:10.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:10.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:10 smithi186 ceph-mon[33438]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T05:02:10.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:10 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:02:10.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:10.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:10 smithi012 ceph-mon[29863]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T05:02:10.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:10 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:02:10.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:10.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:10 smithi012 ceph-mon[41141]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T05:02:10.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:10 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T05:02:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:02:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:11 smithi186 ceph-mon[33438]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T05:02:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:11 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]: dispatch 2024-03-26T05:02:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:11 smithi186 ceph-mon[33438]: pgmap v1165: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:11.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:02:11.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[29863]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T05:02:11.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]: dispatch 2024-03-26T05:02:11.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[29863]: pgmap v1165: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:11.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T05:02:11.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[41141]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T05:02:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]: dispatch 2024-03-26T05:02:11.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:11 smithi012 ceph-mon[41141]: pgmap v1165: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:12.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]': finished 2024-03-26T05:02:12.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:12 smithi186 ceph-mon[33438]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T05:02:12.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:12.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:12 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]: dispatch 2024-03-26T05:02:12.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]': finished 2024-03-26T05:02:12.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[29863]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T05:02:12.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:12.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]: dispatch 2024-03-26T05:02:12.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]': finished 2024-03-26T05:02:12.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[41141]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T05:02:12.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:12.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:12 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-148"}]: dispatch 2024-03-26T05:02:13.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:02:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:02:13.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[29863]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T05:02:13.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:13.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:13.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[29863]: pgmap v1168: 172 pgs: 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:02:13.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[29863]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T05:02:13.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:13.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[41141]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T05:02:13.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:13.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:13.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[41141]: pgmap v1168: 172 pgs: 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:02:13.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[41141]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T05:02:13.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:13 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:13.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:13 smithi186 ceph-mon[33438]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T05:02:13.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:13.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:13 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:13.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:13 smithi186 ceph-mon[33438]: pgmap v1168: 172 pgs: 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:02:13.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:13 smithi186 ceph-mon[33438]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T05:02:13.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:13 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:15 smithi186 ceph-mon[33438]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T05:02:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:15 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-150"}]: dispatch 2024-03-26T05:02:15.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:15 smithi186 ceph-mon[33438]: pgmap v1171: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:02:15.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:15.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[29863]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T05:02:15.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:15.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-150"}]: dispatch 2024-03-26T05:02:15.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[29863]: pgmap v1171: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:02:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[41141]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T05:02:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-150"}]: dispatch 2024-03-26T05:02:15.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:15 smithi012 ceph-mon[41141]: pgmap v1171: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T05:02:16.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:16 smithi186 ceph-mon[33438]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T05:02:16.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:16 smithi012 ceph-mon[29863]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T05:02:16.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:16 smithi012 ceph-mon[41141]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T05:02:17.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:02:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:02:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:17.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:17 smithi012 ceph-mon[29863]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T05:02:17.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:17 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:17.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:17 smithi012 ceph-mon[29863]: pgmap v1174: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:02:17.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:17 smithi012 ceph-mon[41141]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T05:02:17.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:17 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:17.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:17 smithi012 ceph-mon[41141]: pgmap v1174: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:02:17.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:17 smithi186 ceph-mon[33438]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T05:02:17.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:17 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:17.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:17 smithi186 ceph-mon[33438]: pgmap v1174: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[29863]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-152"}]: dispatch 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[29863]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:18.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[41141]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T05:02:18.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:18.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-152"}]: dispatch 2024-03-26T05:02:18.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:18 smithi012 ceph-mon[41141]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:18.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:18.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:18 smithi186 ceph-mon[33438]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T05:02:18.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:18.956 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-152"}]: dispatch 2024-03-26T05:02:18.957 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:18 smithi186 ceph-mon[33438]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[41141]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[41141]: pgmap v1177: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[41141]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:19.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[29863]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T05:02:19.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[29863]: pgmap v1177: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:02:19.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:19.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:19.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[29863]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T05:02:19.899 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:19.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:19 smithi186 ceph-mon[33438]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T05:02:19.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:19 smithi186 ceph-mon[33438]: pgmap v1177: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T05:02:19.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:19 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:19.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:19 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:19.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:19 smithi186 ceph-mon[33438]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T05:02:19.958 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T05:02:21.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:21.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[29863]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T05:02:21.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-111","var": "dedup_tier","val": "test-rados-api-smithi012-78824-154"}]: dispatch 2024-03-26T05:02:21.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:21.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-154"}]: dispatch 2024-03-26T05:02:21.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[29863]: pgmap v1180: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:21.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:21.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[41141]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T05:02:21.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-111","var": "dedup_tier","val": "test-rados-api-smithi012-78824-154"}]: dispatch 2024-03-26T05:02:21.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:21.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-154"}]: dispatch 2024-03-26T05:02:21.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:21 smithi012 ceph-mon[41141]: pgmap v1180: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:21.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi012-78824-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T05:02:21.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:21 smithi186 ceph-mon[33438]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T05:02:21.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi012-78824-111","var": "dedup_tier","val": "test-rados-api-smithi012-78824-154"}]: dispatch 2024-03-26T05:02:21.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:21.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:21 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi012-78824-111", "tierpool": "test-rados-api-smithi012-78824-154"}]: dispatch 2024-03-26T05:02:21.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:21 smithi186 ceph-mon[33438]: pgmap v1180: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:22.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:22 smithi012 ceph-mon[29863]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T05:02:22.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:22 smithi012 ceph-mon[41141]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T05:02:22.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:22 smithi186 ceph-mon[33438]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T05:02:23.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:02:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:02:23.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[29863]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T05:02:23.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:23.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:23.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:23.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[29863]: pgmap v1183: 164 pgs: 164 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:23.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[41141]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T05:02:23.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:23.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:23.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:23.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:23 smithi012 ceph-mon[41141]: pgmap v1183: 164 pgs: 164 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:23.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:23 smithi186 ceph-mon[33438]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T05:02:23.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:23 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:23.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:23.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:23 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:23.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:23 smithi186 ceph-mon[33438]: pgmap v1183: 164 pgs: 164 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:24.255 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7892 ms) 2024-03-26T05:02:24.255 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-26T05:02:24.255 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13067 ms) 2024-03-26T05:02:24.255 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-26T05:02:24.255 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8049 ms) 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13104 ms) 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7807 ms) 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8064 ms) 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: ok, hit_set contains 337:602f83fe:::foo:head 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8101 ms) 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,0 2024-03-26T05:02:24.256 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: first is 1711429259 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,1711429265,1711429266,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,1711429265,1711429266,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,1711429265,1711429266,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,1711429265,1711429266,1711429268,1711429269,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,1711429265,1711429266,1711429268,1711429269,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429259,1711429262,1711429263,1711429265,1711429266,1711429268,1711429269,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: got ls 1711429263,1711429265,1711429266,1711429268,1711429269,1711429271,1711429272,0 2024-03-26T05:02:24.257 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: first now 1711429263, trimmed 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20156 ms) 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: foo0 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14056 ms) 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17996 ms) 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22993 ms) 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5041 ms) 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-26T05:02:24.258 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3010 ms) 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3019 ms) 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3016 ms) 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (230807 ms total) 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (866783 ms total) 2024-03-26T05:02:24.259 INFO:tasks.workunit.client.0.smithi012.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-26T05:02:24.262 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78526 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78526 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78805 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78805 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79223 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79223 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78970 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78970 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78851 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78851 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78601 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78601 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78899 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78899 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79402 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79402 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79537 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79537 2024-03-26T05:02:24.264 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78627 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78627 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79237 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79237 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79110 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79110 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78499 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78499 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78563 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78563 2024-03-26T05:02:24.265 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79151 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79151 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78490 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78490 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78510 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78510 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79593 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79593 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78816 2024-03-26T05:02:24.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78816 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79706 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79706 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78867 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78867 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79352 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79352 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79767 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79767 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78829 2024-03-26T05:02:24.267 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78829 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79205 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79205 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79573 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79573 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79326 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79326 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79661 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79661 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79089 2024-03-26T05:02:24.268 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79089 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78746 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78746 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79718 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79718 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79006 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79006 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78483 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78483 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78544 2024-03-26T05:02:24.269 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78544 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78946 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78946 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=78586 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 78586 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79041 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79041 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79075 2024-03-26T05:02:24.270 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79075 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79561 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79561 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ for t in "${!pids[@]}" 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=79799 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 79799 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ exit 0 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ cleanup 2024-03-26T05:02:24.271 INFO:tasks.workunit.client.0.smithi012.stderr:+ pkill -P 78477 2024-03-26T05:02:24.277 INFO:tasks.workunit.client.0.smithi012.stderr:+ true 2024-03-26T05:02:24.278 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T05:02:24.278 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T05:02:24.311 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-26T05:02:24.312 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-26T05:02:24.313 DEBUG:teuthology.orchestra.run.smithi012:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-26T05:02:24.371 INFO:tasks.workunit.client.0.smithi012.stderr:++ uuidgen 2024-03-26T05:02:24.385 INFO:tasks.workunit.client.0.smithi012.stderr:+ p=0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:02:24.385 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool create 0fa966d1-3729-426d-8271-2d6b10038686 12 2024-03-26T05:02:24.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 -- 172.21.15.12:0/611055396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4101580 msgr2=0x7f4db4101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/611055396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4101580 0x7f4db4101a00 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009900 tx=0x7f4da802bc30 comp rx=0 tx=0).stop 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 -- 172.21.15.12:0/611055396 shutdown_connections 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/611055396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4db4101f40 0x7f4db410e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/611055396 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4101580 0x7f4db4101a00 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/611055396 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4107640 0x7f4db4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 -- 172.21.15.12:0/611055396 >> 172.21.15.12:0/611055396 conn(0x7f4db40fd050 msgr2=0x7f4db40ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 -- 172.21.15.12:0/611055396 shutdown_connections 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 -- 172.21.15.12:0/611055396 wait complete. 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 Processor -- start 2024-03-26T05:02:24.474 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.479+0000 7f4dbabc7640 1 -- start start 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 0x7f4db4073890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4db4101f40 0x7f4db4073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4107640 0x7f4db4074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41a54f0 con 0x7f4db4107640 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41a5660 con 0x7f4db4101f40 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41a5820 con 0x7f4db4101580 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 0x7f4db4073890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 0x7f4db4073890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:37568/0 (socket says 172.21.15.12:37568) 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 -- 172.21.15.12:0/1755654637 learned_addr learned my addr 172.21.15.12:0/1755654637 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db9164640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4107640 0x7f4db4074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 -- 172.21.15.12:0/1755654637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4db4101f40 msgr2=0x7f4db4073dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4db4101f40 0x7f4db4073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 -- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4107640 msgr2=0x7f4db4074310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:24.475 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4107640 0x7f4db4074310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 -- 172.21.15.12:0/1755654637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac009840 con 0x7f4db4101580 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db8963640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 0x7f4db4073890 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009d50 tx=0x7f4dac00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac003b70 con 0x7f4db4101580 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4dac003cd0 con 0x7f4db4101580 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da8009560 con 0x7f4db4101580 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.480+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac010730 con 0x7f4db4101580 2024-03-26T05:02:24.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.481+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db41a5e80 con 0x7f4db4101580 2024-03-26T05:02:24.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.482+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40688c0 con 0x7f4db4101580 2024-03-26T05:02:24.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.482+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f4dac004020 con 0x7f4db4101580 2024-03-26T05:02:24.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.482+0000 7f4db1ffb640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f4d9c06da30 0x7f4d9c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:24.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.482+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(761..761 src has 259..761) v4 ==== 8824+0+0 (secure 0 0 0) 0x7f4dac014070 con 0x7f4db4101580 2024-03-26T05:02:24.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.482+0000 7f4db3fff640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f4d9c06da30 0x7f4d9c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:24.477 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.483+0000 7f4db3fff640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f4d9c06da30 0x7f4d9c06fef0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f4da8036040 tx=0x7f4da8000bc0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:24.480 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.487+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4dac057590 con 0x7f4db4101580 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[41141]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[41141]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[29863]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T05:02:24.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:24.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:24.649 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:24 smithi012 ceph-mon[29863]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T05:02:24.661 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:24.669+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12} v 0) v1 -- 0x7f4db4110bd0 con 0x7f4db4101580 2024-03-26T05:02:24.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:24.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:24 smithi186 ceph-mon[33438]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T05:02:24.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]: dispatch 2024-03-26T05:02:24.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:24 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/613909254' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi012-78824-111"}]': finished 2024-03-26T05:02:24.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:24 smithi186 ceph-mon[33438]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T05:02:25.261 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.268+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]=0 pool '0fa966d1-3729-426d-8271-2d6b10038686' created v762) v1 ==== 176+0+0 (secure 0 0 0) 0x7f4dac055e60 con 0x7f4db4101580 2024-03-26T05:02:25.372 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.379+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12} v 0) v1 -- 0x7f4db4110d60 con 0x7f4db4101580 2024-03-26T05:02:25.373 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.380+0000 7f4db1ffb640 1 -- 172.21.15.12:0/1755654637 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]=0 pool '0fa966d1-3729-426d-8271-2d6b10038686' already exists v762) v1 ==== 183+0+0 (secure 0 0 0) 0x7f4dac055e60 con 0x7f4db4101580 2024-03-26T05:02:25.373 INFO:tasks.workunit.client.0.smithi012.stderr:pool '0fa966d1-3729-426d-8271-2d6b10038686' already exists 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f4d9c06da30 msgr2=0x7f4d9c06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f4d9c06da30 0x7f4d9c06fef0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f4da8036040 tx=0x7f4da8000bc0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 msgr2=0x7f4db4073890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 0x7f4db4073890 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009d50 tx=0x7f4dac00c6a0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 shutdown_connections 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db4107640 0x7f4db4074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4db4101f40 0x7f4db4073dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f4d9c06da30 0x7f4d9c06fef0 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.375 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 --2- 172.21.15.12:0/1755654637 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f4db4101580 0x7f4db4073890 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 >> 172.21.15.12:0/1755654637 conn(0x7f4db40fd050 msgr2=0x7f4db40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:25.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 shutdown_connections 2024-03-26T05:02:25.376 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.382+0000 7f4dbabc7640 1 -- 172.21.15.12:0/1755654637 wait complete. 2024-03-26T05:02:25.384 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_objects 10 2024-03-26T05:02:25.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 -- 172.21.15.12:0/994127640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 msgr2=0x7f27ec107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:25.469 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 --2- 172.21.15.12:0/994127640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec107a10 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009cd0 tx=0x7f27e402c0f0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 -- 172.21.15.12:0/994127640 shutdown_connections 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 --2- 172.21.15.12:0/994127640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27ec101f30 0x7f27ec10e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 --2- 172.21.15.12:0/994127640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101570 0x7f27ec1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 --2- 172.21.15.12:0/994127640 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec107a10 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 -- 172.21.15.12:0/994127640 >> 172.21.15.12:0/994127640 conn(0x7f27ec0fd020 msgr2=0x7f27ec0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.474+0000 7f27f4501640 1 -- 172.21.15.12:0/994127640 shutdown_connections 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 -- 172.21.15.12:0/994127640 wait complete. 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 Processor -- start 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 -- start start 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27ec101570 0x7f27ec1971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 0x7f27ec197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:25.470 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec19c0a0 con 0x7f27ec101f30 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec19c210 con 0x7f27ec101570 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f4501640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec19c3d0 con 0x7f27ec107630 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f1a9c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 0x7f27ec197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f2a9e640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f1a9c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 0x7f27ec197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36870/0 (socket says 172.21.15.12:36870) 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.475+0000 7f27f1a9c640 1 -- 172.21.15.12:0/150768645 learned_addr learned my addr 172.21.15.12:0/150768645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f229d640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27ec101570 0x7f27ec1971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f1a9c640 1 -- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 msgr2=0x7f27ec19baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f1a9c640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec19baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f1a9c640 1 -- 172.21.15.12:0/150768645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27ec101570 msgr2=0x7f27ec1971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f1a9c640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27ec101570 0x7f27ec1971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f1a9c640 1 -- 172.21.15.12:0/150768645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e0009490 con 0x7f27ec101f30 2024-03-26T05:02:25.471 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f2a9e640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec19baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:02:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f1a9c640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 0x7f27ec197720 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f27e000ea60 tx=0x7f27e000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000cde0 con 0x7f27ec101f30 2024-03-26T05:02:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e4009900 con 0x7f27ec101f30 2024-03-26T05:02:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f27e00044a0 con 0x7f27ec101f30 2024-03-26T05:02:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0010430 con 0x7f27ec101f30 2024-03-26T05:02:25.472 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.476+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec1a3d50 con 0x7f27ec101f30 2024-03-26T05:02:25.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.477+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec068470 con 0x7f27ec101f30 2024-03-26T05:02:25.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.478+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f27e0003f80 con 0x7f27ec101f30 2024-03-26T05:02:25.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.478+0000 7f27eb7fe640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f27d406da30 0x7f27d406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:25.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.478+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(762..762 src has 259..762) v4 ==== 9199+0+0 (secure 0 0 0) 0x7f27e0014070 con 0x7f27ec101f30 2024-03-26T05:02:25.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.478+0000 7f27f229d640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f27d406da30 0x7f27d406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:25.473 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.479+0000 7f27f229d640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f27d406da30 0x7f27d406fef0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f27e4006010 tx=0x7f27e4005c30 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:25.476 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.483+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f27e00571f0 con 0x7f27ec101f30 2024-03-26T05:02:25.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:25 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1755654637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:25 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:25.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:25 smithi012 ceph-mon[29863]: pgmap v1186: 164 pgs: 164 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:25.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:25 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1755654637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:25.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:25 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:25.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:25 smithi012 ceph-mon[41141]: pgmap v1186: 164 pgs: 164 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:25.660 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:25.667+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f27ec110b70 con 0x7f27ec101f30 2024-03-26T05:02:25.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:25 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1755654637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:25.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:25 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:25.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:25 smithi186 ceph-mon[33438]: pgmap v1186: 164 pgs: 164 active+clean; 582 KiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:26.258 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:26.265+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v763) v1 ==== 223+0+0 (secure 0 0 0) 0x7f27e005a8b0 con 0x7f27ec101f30 2024-03-26T05:02:26.379 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:26.387+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f27ec198530 con 0x7f27ec101f30 2024-03-26T05:02:26.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]': finished 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[29863]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1755654637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]': finished 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[41141]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1755654637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:26.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:02:26.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]': finished 2024-03-26T05:02:26.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:26 smithi186 ceph-mon[33438]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T05:02:26.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1755654637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:26 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pg_num": 12}]: dispatch 2024-03-26T05:02:26.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:26 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:02:27.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:02:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:02:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:27.274 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.281+0000 7f27eb7fe640 1 -- 172.21.15.12:0/150768645 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v764) v1 ==== 223+0+0 (secure 0 0 0) 0x7f27e005aa90 con 0x7f27ec101f30 2024-03-26T05:02:27.274 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_objects = 10 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:02:27.279 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.285+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f27d406da30 msgr2=0x7f27d406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:27.279 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f27d406da30 0x7f27d406fef0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f27e4006010 tx=0x7f27e4005c30 comp rx=0 tx=0).stop 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 msgr2=0x7f27ec197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 0x7f27ec197720 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f27e000ea60 tx=0x7f27e000ef20 comp rx=0 tx=0).stop 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 shutdown_connections 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f27d406da30 0x7f27d406fef0 unknown :-1 s=CLOSED pgs=3509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f27ec107630 0x7f27ec19baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27ec101f30 0x7f27ec197720 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 --2- 172.21.15.12:0/150768645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27ec101570 0x7f27ec1971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.286+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 >> 172.21.15.12:0/150768645 conn(0x7f27ec0fd020 msgr2=0x7f27ec108330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.287+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 shutdown_connections 2024-03-26T05:02:27.280 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.287+0000 7f27f4501640 1 -- 172.21.15.12:0/150768645 wait complete. 2024-03-26T05:02:27.291 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool application enable 0fa966d1-3729-426d-8271-2d6b10038686 rados 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 -- 172.21.15.12:0/3193404999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 msgr2=0x7f06b810d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 --2- 172.21.15.12:0/3193404999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b810d310 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009ca0 tx=0x7f06b002c0f0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 -- 172.21.15.12:0/3193404999 shutdown_connections 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 --2- 172.21.15.12:0/3193404999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b810d310 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 --2- 172.21.15.12:0/3193404999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06b8100490 0x7f06b8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 --2- 172.21.15.12:0/3193404999 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 0x7f06b8106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.383 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 -- 172.21.15.12:0/3193404999 >> 172.21.15.12:0/3193404999 conn(0x7f06b8077670 msgr2=0x7f06b8077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 -- 172.21.15.12:0/3193404999 shutdown_connections 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 -- 172.21.15.12:0/3193404999 wait complete. 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 Processor -- start 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.388+0000 7f06bd580640 1 -- start start 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06b8100490 0x7f06b8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 0x7f06b81a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8198c10 con 0x7f06b8100e50 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8198d80 con 0x7f06b8100490 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b81a34f0 con 0x7f06b8106550 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b67fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36882/0 (socket says 172.21.15.12:36882) 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b67fc640 1 -- 172.21.15.12:0/1267338021 learned_addr learned my addr 172.21.15.12:0/1267338021 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:02:27.384 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 0x7f06b81a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b6ffd640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06b8100490 0x7f06b8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 -- 172.21.15.12:0/1267338021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06b8100490 msgr2=0x7f06b8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06b8100490 0x7f06b8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 -- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 msgr2=0x7f06b8198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b8198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 -- 172.21.15.12:0/1267338021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8009490 con 0x7f06b8106550 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b67fc640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06b77fe640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 0x7f06b81a2f60 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f06b002c700 tx=0x7f06b0032f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0034c30 con 0x7f06b8106550 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f06b0032980 con 0x7f06b8106550 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.389+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b0009900 con 0x7f06b8106550 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.390+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b00045b0 con 0x7f06b8106550 2024-03-26T05:02:27.385 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.390+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b81a3a60 con 0x7f06b8106550 2024-03-26T05:02:27.386 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.391+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b8101bb0 con 0x7f06b8106550 2024-03-26T05:02:27.386 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.391+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f06b003b040 con 0x7f06b8106550 2024-03-26T05:02:27.387 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.392+0000 7f069ffff640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f06a006da30 0x7f06a006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:27.387 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.392+0000 7f06b6ffd640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f06a006da30 0x7f06a006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:27.387 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.392+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(764..764 src has 259..764) v4 ==== 9199+0+0 (secure 0 0 0) 0x7f06b00b0470 con 0x7f06b8106550 2024-03-26T05:02:27.387 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.392+0000 7f06b6ffd640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f06a006da30 0x7f06a006fef0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009ce0 tx=0x7f06a8005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:27.389 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.396+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f06b0079250 con 0x7f06b8106550 2024-03-26T05:02:27.572 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:27.579+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"} v 0) v1 -- 0x7f06b810fa60 con 0x7f06b8106550 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[29863]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[29863]: pgmap v1189: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[41141]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T05:02:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:02:27.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:27 smithi012 ceph-mon[41141]: pgmap v1189: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:27.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:02:27.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:27 smithi186 ceph-mon[33438]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T05:02:27.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:27 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:02:27.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:27 smithi186 ceph-mon[33438]: pgmap v1189: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:28.284 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:28.291+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]=0 enabled application 'rados' on pool '0fa966d1-3729-426d-8271-2d6b10038686' v765) v1 ==== 213+0+0 (secure 0 0 0) 0x7f06b007c910 con 0x7f06b8106550 2024-03-26T05:02:28.396 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:28.403+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"} v 0) v1 -- 0x7f06b810fc70 con 0x7f06b8106550 2024-03-26T05:02:28.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[29863]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1267338021' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[41141]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1267338021' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:28.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:28 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:28.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/150768645' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:02:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:28 smithi186 ceph-mon[33438]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T05:02:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:28 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1267338021' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:28.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:28 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.293 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.300+0000 7f069ffff640 1 -- 172.21.15.12:0/1267338021 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]=0 enabled application 'rados' on pool '0fa966d1-3729-426d-8271-2d6b10038686' v766) v1 ==== 213+0+0 (secure 0 0 0) 0x7f06b007caf0 con 0x7f06b8106550 2024-03-26T05:02:29.293 INFO:tasks.workunit.client.0.smithi012.stderr:enabled application 'rados' on pool '0fa966d1-3729-426d-8271-2d6b10038686' 2024-03-26T05:02:29.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f06a006da30 msgr2=0x7f06a006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:29.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f06a006da30 0x7f06a006fef0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009ce0 tx=0x7f06a8005f90 comp rx=0 tx=0).stop 2024-03-26T05:02:29.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 msgr2=0x7f06b81a2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:29.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 0x7f06b81a2f60 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f06b002c700 tx=0x7f06b0032f80 comp rx=0 tx=0).stop 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 shutdown_connections 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f06a006da30 0x7f06a006fef0 unknown :-1 s=CLOSED pgs=3510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f06b8106550 0x7f06b81a2f60 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06b8100e50 0x7f06b8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 --2- 172.21.15.12:0/1267338021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06b8100490 0x7f06b8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 >> 172.21.15.12:0/1267338021 conn(0x7f06b8077670 msgr2=0x7f06b8104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.303+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 shutdown_connections 2024-03-26T05:02:29.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:29.304+0000 7f06bd580640 1 -- 172.21.15.12:0/1267338021 wait complete. 2024-03-26T05:02:29.308 INFO:tasks.workunit.client.0.smithi012.stderr:++ seq 1 10 2024-03-26T05:02:29.322 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.322 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj1 /etc/passwd 2024-03-26T05:02:29.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.363 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj2 /etc/passwd 2024-03-26T05:02:29.399 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.399 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj3 /etc/passwd 2024-03-26T05:02:29.435 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.436 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj4 /etc/passwd 2024-03-26T05:02:29.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.470 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj5 /etc/passwd 2024-03-26T05:02:29.504 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.505 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj6 /etc/passwd 2024-03-26T05:02:29.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj7 /etc/passwd 2024-03-26T05:02:29.574 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.574 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj8 /etc/passwd 2024-03-26T05:02:29.609 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.609 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj9 /etc/passwd 2024-03-26T05:02:29.643 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:02:29.643 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put obj10 /etc/passwd 2024-03-26T05:02:29.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]': finished 2024-03-26T05:02:29.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[29863]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/1267338021' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[29863]: pgmap v1192: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]': finished 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[41141]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/1267338021' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:29 smithi012 ceph-mon[41141]: pgmap v1192: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:29.677 INFO:tasks.workunit.client.0.smithi012.stderr:+ sleep 30 2024-03-26T05:02:29.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]': finished 2024-03-26T05:02:29.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:29 smithi186 ceph-mon[33438]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T05:02:29.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:29 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/1267338021' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:29 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]: dispatch 2024-03-26T05:02:29.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:29 smithi186 ceph-mon[33438]: pgmap v1192: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:30.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:30 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:30.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:30 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]': finished 2024-03-26T05:02:30.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:30 smithi012 ceph-mon[29863]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T05:02:30.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:30 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:30.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:30 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]': finished 2024-03-26T05:02:30.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:30 smithi012 ceph-mon[41141]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T05:02:30.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:30 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:02:30.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:30 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "app": "rados"}]': finished 2024-03-26T05:02:30.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:30 smithi186 ceph-mon[33438]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T05:02:31.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:31 smithi012 ceph-mon[29863]: pgmap v1194: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:31.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:31 smithi012 ceph-mon[41141]: pgmap v1194: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:31.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:31 smithi186 ceph-mon[33438]: pgmap v1194: 176 pgs: 12 creating+peering, 164 active+clean; 582 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:02:33.149 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:02:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:02:33.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:33.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:33 smithi186 ceph-mon[33438]: pgmap v1195: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-26T05:02:33.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:33.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:33 smithi012 ceph-mon[29863]: pgmap v1195: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-26T05:02:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:33 smithi012 ceph-mon[41141]: pgmap v1195: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 20 KiB/s wr, 2 op/s 2024-03-26T05:02:34.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:34.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:34.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:35.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:35 smithi012 ceph-mon[29863]: pgmap v1196: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 657 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-26T05:02:35.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:35 smithi012 ceph-mon[41141]: pgmap v1196: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 657 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-26T05:02:35.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:35 smithi186 ceph-mon[33438]: pgmap v1196: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 657 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-26T05:02:37.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:02:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:02:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:37.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:37 smithi186 ceph-mon[33438]: pgmap v1197: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-03-26T05:02:37.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:37 smithi012 ceph-mon[29863]: pgmap v1197: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-03-26T05:02:37.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:37 smithi012 ceph-mon[41141]: pgmap v1197: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-03-26T05:02:38.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:38 smithi186 ceph-mon[33438]: pool '0fa966d1-3729-426d-8271-2d6b10038686' is full (reached quota's max_objects: 10) 2024-03-26T05:02:38.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:38 smithi186 ceph-mon[33438]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:02:38.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:38 smithi186 ceph-mon[33438]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T05:02:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:38 smithi012 ceph-mon[29863]: pool '0fa966d1-3729-426d-8271-2d6b10038686' is full (reached quota's max_objects: 10) 2024-03-26T05:02:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:38 smithi012 ceph-mon[29863]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:02:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:38 smithi012 ceph-mon[29863]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T05:02:38.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:38 smithi012 ceph-mon[41141]: pool '0fa966d1-3729-426d-8271-2d6b10038686' is full (reached quota's max_objects: 10) 2024-03-26T05:02:38.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:38 smithi012 ceph-mon[41141]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:02:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:38 smithi012 ceph-mon[41141]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T05:02:39.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:39 smithi186 ceph-mon[33438]: pgmap v1199: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T05:02:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:39 smithi012 ceph-mon[29863]: pgmap v1199: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T05:02:39.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:39 smithi012 ceph-mon[41141]: pgmap v1199: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T05:02:41.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:41 smithi186 ceph-mon[33438]: pgmap v1200: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:02:41.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:41 smithi012 ceph-mon[29863]: pgmap v1200: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:02:41.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:41 smithi012 ceph-mon[41141]: pgmap v1200: 176 pgs: 176 active+clean; 698 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:02:43.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:02:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:02:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:43 smithi186 ceph-mon[33438]: pgmap v1201: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:43.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:43.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:43 smithi012 ceph-mon[29863]: pgmap v1201: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:43.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:43.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:43 smithi012 ceph-mon[41141]: pgmap v1201: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:45 smithi186 ceph-mon[33438]: pgmap v1202: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:45.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:45 smithi012 ceph-mon[29863]: pgmap v1202: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:45.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:45 smithi012 ceph-mon[41141]: pgmap v1202: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:47.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:02:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:02:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:47.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:47 smithi186 ceph-mon[33438]: pgmap v1203: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:47 smithi012 ceph-mon[29863]: pgmap v1203: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:47 smithi012 ceph-mon[41141]: pgmap v1203: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:02:49.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:49 smithi186 ceph-mon[33438]: pgmap v1204: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-26T05:02:49.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:49 smithi012 ceph-mon[29863]: pgmap v1204: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-26T05:02:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:49 smithi012 ceph-mon[41141]: pgmap v1204: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-26T05:02:50.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:50 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:50.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:50 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:50.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:50 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:50.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:50 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:50.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:50 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:02:50.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:50 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:02:51.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:51 smithi186 ceph-mon[33438]: pgmap v1205: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:51.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:51 smithi012 ceph-mon[29863]: pgmap v1205: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:51.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:51 smithi012 ceph-mon[41141]: pgmap v1205: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:53.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:02:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:02:53.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:53.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:53 smithi186 ceph-mon[33438]: pgmap v1206: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:53.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:53.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:53 smithi012 ceph-mon[29863]: pgmap v1206: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:02:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:53 smithi012 ceph-mon[41141]: pgmap v1206: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:55.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:55 smithi186 ceph-mon[33438]: pgmap v1207: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:55.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:55 smithi012 ceph-mon[29863]: pgmap v1207: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:55.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:55 smithi012 ceph-mon[41141]: pgmap v1207: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:57.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:02:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:02:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:02:57.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:57 smithi186 ceph-mon[33438]: pgmap v1208: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:57.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:57 smithi012 ceph-mon[29863]: pgmap v1208: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:57.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:57 smithi012 ceph-mon[41141]: pgmap v1208: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:02:59.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:02:59 smithi186 ceph-mon[33438]: pgmap v1209: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:59.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:02:59 smithi012 ceph-mon[29863]: pgmap v1209: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:59.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:02:59 smithi012 ceph-mon[41141]: pgmap v1209: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:02:59.678 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=149503 2024-03-26T05:02:59.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_objects 100 2024-03-26T05:02:59.679 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put onemore /etc/passwd 2024-03-26T05:02:59.776 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 -- 172.21.15.12:0/1439056755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 msgr2=0x7f5b1810e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:59.776 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 --2- 172.21.15.12:0/1439056755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b1810e400 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f5b10009cd0 tx=0x7f5b1002c0f0 comp rx=0 tx=0).stop 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 -- 172.21.15.12:0/1439056755 shutdown_connections 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 --2- 172.21.15.12:0/1439056755 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b1810e400 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 --2- 172.21.15.12:0/1439056755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18101580 0x7f5b18101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 --2- 172.21.15.12:0/1439056755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18107640 0x7f5b18107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 -- 172.21.15.12:0/1439056755 >> 172.21.15.12:0/1439056755 conn(0x7f5b180fd090 msgr2=0x7f5b180ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 -- 172.21.15.12:0/1439056755 shutdown_connections 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.781+0000 7f5b1e488640 1 -- 172.21.15.12:0/1439056755 wait complete. 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 Processor -- start 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 -- start start 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18101580 0x7f5b181970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b181975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 0x7f5b1819b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:59.777 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1819bf20 con 0x7f5b18107640 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1819c090 con 0x7f5b18101580 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1e488640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1819c250 con 0x7f5b18101f40 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1ca25640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 0x7f5b1819b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b177fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b181975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1ca25640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 0x7f5b1819b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48234/0 (socket says 172.21.15.12:48234) 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.782+0000 7f5b1ca25640 1 -- 172.21.15.12:0/3141040102 learned_addr learned my addr 172.21.15.12:0/3141040102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b17fff640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18101580 0x7f5b181970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b177fe640 1 -- 172.21.15.12:0/3141040102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18101580 msgr2=0x7f5b181970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b177fe640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18101580 0x7f5b181970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b177fe640 1 -- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 msgr2=0x7f5b1819b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b177fe640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 0x7f5b1819b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:02:59.778 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b177fe640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c009490 con 0x7f5b18101f40 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b17fff640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18101580 0x7f5b181970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b1ca25640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 0x7f5b1819b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b177fe640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b181975f0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00e9f0 tx=0x7f5b0c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c00cd80 con 0x7f5b18101f40 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b10009900 con 0x7f5b18101f40 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5b0c0044a0 con 0x7f5b18101f40 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c003e70 con 0x7f5b18101f40 2024-03-26T05:02:59.779 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.783+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b181a3bd0 con 0x7f5b18101f40 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.784+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b180688c0 con 0x7f5b18101f40 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.785+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f5b0c004000 con 0x7f5b18101f40 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.785+0000 7f5b157fa640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f5b0006da30 0x7f5b0006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.785+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(767..767 src has 259..767) v4 ==== 9212+0+0 (secure 0 0 0) 0x7f5b0c014070 con 0x7f5b18101f40 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.785+0000 7f5b17fff640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f5b0006da30 0x7f5b0006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.785+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=768}) v3 -- 0x7f5b00079740 con 0x7f5b18101f40 2024-03-26T05:02:59.780 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.786+0000 7f5b17fff640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f5b0006da30 0x7f5b0006fef0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f5b080058c0 tx=0x7f5b08005850 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:02:59.782 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.790+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5b0c057160 con 0x7f5b18101f40 2024-03-26T05:02:59.965 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:02:59.973+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f5b18110b40 con 0x7f5b18101f40 2024-03-26T05:03:00.162 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:00.169+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 7 ==== osd_map(768..768 src has 259..768) v4 ==== 628+0+0 (secure 0 0 0) 0x7f5b0c05b5c0 con 0x7f5b18101f40 2024-03-26T05:03:00.162 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:00.169+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=769}) v3 -- 0x7f5b0007a270 con 0x7f5b18101f40 2024-03-26T05:03:00.164 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:00.172+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v768) v1 ==== 225+0+0 (secure 0 0 0) 0x7f5b0c05a820 con 0x7f5b18101f40 2024-03-26T05:03:00.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:00.305+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f5b18110cd0 con 0x7f5b18101f40 2024-03-26T05:03:00.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:00 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3141040102' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:00.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:00 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:00.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:00 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3141040102' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:00.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:00 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:00.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:00 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3141040102' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:00.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:00 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.169 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.176+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 9 ==== osd_map(769..769 src has 259..769) v4 ==== 628+0+0 (secure 0 0 0) 0x7f5b0c05b5c0 con 0x7f5b18101f40 2024-03-26T05:03:01.169 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.176+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=770}) v3 -- 0x7f5b0007a660 con 0x7f5b18101f40 2024-03-26T05:03:01.172 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.180+0000 7f5b157fa640 1 -- 172.21.15.12:0/3141040102 <== mon.2 v2:172.21.15.12:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v769) v1 ==== 225+0+0 (secure 0 0 0) 0x7f5b0c05a820 con 0x7f5b18101f40 2024-03-26T05:03:01.173 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_objects = 100 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f5b0006da30 msgr2=0x7f5b0006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f5b0006da30 0x7f5b0006fef0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f5b080058c0 tx=0x7f5b08005850 comp rx=0 tx=0).stop 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 msgr2=0x7f5b181975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b181975f0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00e9f0 tx=0x7f5b0c00eeb0 comp rx=0 tx=0).stop 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 shutdown_connections 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b18107640 0x7f5b1819b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f5b0006da30 0x7f5b0006fef0 unknown :-1 s=CLOSED pgs=3522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:01.176 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f5b18101f40 0x7f5b181975f0 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:01.177 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 --2- 172.21.15.12:0/3141040102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b18101580 0x7f5b181970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:01.177 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 >> 172.21.15.12:0/3141040102 conn(0x7f5b180fd090 msgr2=0x7f5b18108340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:01.177 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 shutdown_connections 2024-03-26T05:03:01.177 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:01.183+0000 7f5b1e488640 1 -- 172.21.15.12:0/3141040102 wait complete. 2024-03-26T05:03:01.185 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 149503 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3141040102' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:03:01.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:01.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:01 smithi186 ceph-mon[33438]: pgmap v1211: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3141040102' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:01.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[29863]: pgmap v1211: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3141040102' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:01.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:01 smithi012 ceph-mon[41141]: pgmap v1211: 176 pgs: 176 active+clean; 698 KiB data, 796 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:03:02.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:02 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:03:02.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:02 smithi186 ceph-mon[33438]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T05:03:02.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:02 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:03:02.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:02 smithi012 ceph-mon[29863]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T05:03:02.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:02 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "100"}]': finished 2024-03-26T05:03:02.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:02 smithi012 ceph-mon[41141]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T05:03:02.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T05:03:02.814 INFO:tasks.workunit.client.0.smithi012.stderr:+ true 2024-03-26T05:03:02.815 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put twomore /etc/passwd 2024-03-26T05:03:02.853 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_bytes 100 2024-03-26T05:03:02.942 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 -- 172.21.15.12:0/1520377253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100490 msgr2=0x7f393c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:02.942 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 --2- 172.21.15.12:0/1520377253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100490 0x7f393c100910 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7f3928009930 tx=0x7f392802bc30 comp rx=0 tx=0).stop 2024-03-26T05:03:02.942 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 -- 172.21.15.12:0/1520377253 shutdown_connections 2024-03-26T05:03:02.942 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 --2- 172.21.15.12:0/1520377253 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c100e50 0x7f393c10d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:02.942 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 --2- 172.21.15.12:0/1520377253 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100490 0x7f393c100910 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 --2- 172.21.15.12:0/1520377253 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f393c106550 0x7f393c106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 -- 172.21.15.12:0/1520377253 >> 172.21.15.12:0/1520377253 conn(0x7f393c077670 msgr2=0x7f393c077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 -- 172.21.15.12:0/1520377253 shutdown_connections 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.947+0000 7f3942c9a640 1 -- 172.21.15.12:0/1520377253 wait complete. 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 Processor -- start 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 -- start start 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f393c100490 0x7f393c102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 0x7f393c102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 0x7f393c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1a5490 con 0x7f393c106550 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1a5600 con 0x7f393c100490 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3942c9a640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1a57c0 con 0x7f393c100e50 2024-03-26T05:03:02.943 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 0x7f393c102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3941237640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 0x7f393c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 0x7f393c102640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:41376/0 (socket says 172.21.15.12:41376) 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3941237640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 0x7f393c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48284/0 (socket says 172.21.15.12:48284) 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 -- 172.21.15.12:0/3387510748 learned_addr learned my addr 172.21.15.12:0/3387510748 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3940a36640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f393c100490 0x7f393c102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 -- 172.21.15.12:0/3387510748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f393c100490 msgr2=0x7f393c102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f393c100490 0x7f393c102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 -- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 msgr2=0x7f393c102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 0x7f393c102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f393bfff640 1 -- 172.21.15.12:0/3387510748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934009840 con 0x7f393c100e50 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.948+0000 7f3941237640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 0x7f393c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.949+0000 7f393bfff640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 0x7f393c102640 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f3928005ab0 tx=0x7f39280042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.949+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280044e0 con 0x7f393c100e50 2024-03-26T05:03:02.944 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.949+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3928009560 con 0x7f393c100e50 2024-03-26T05:03:02.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.949+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3928003c80 con 0x7f393c100e50 2024-03-26T05:03:02.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.949+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280053f0 con 0x7f393c100e50 2024-03-26T05:03:02.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.949+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c1a5e10 con 0x7f393c100e50 2024-03-26T05:03:02.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.950+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c0688c0 con 0x7f393c100e50 2024-03-26T05:03:02.946 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.950+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f3928003800 con 0x7f393c100e50 2024-03-26T05:03:02.946 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.951+0000 7f3939ffb640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f392c06da30 0x7f392c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:02.946 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.951+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(770..770 src has 259..770) v4 ==== 9212+0+0 (secure 0 0 0) 0x7f39280b0970 con 0x7f393c100e50 2024-03-26T05:03:02.946 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.951+0000 7f3940a36640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f392c06da30 0x7f392c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:02.946 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.951+0000 7f3940a36640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f392c06da30 0x7f392c06fef0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7f3930005d00 tx=0x7f393000b410 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:02.948 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:02.955+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3928079630 con 0x7f393c100e50 2024-03-26T05:03:03.131 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:03.139+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f393c1a59e0 con 0x7f393c100e50 2024-03-26T05:03:03.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:03:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:03:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: pool '0fa966d1-3729-426d-8271-2d6b10038686' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:03:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:03:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T05:03:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: pgmap v1214: 176 pgs: 176 active+clean; 698 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3387510748' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:03.457 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:03 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: pool '0fa966d1-3729-426d-8271-2d6b10038686' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: pgmap v1214: 176 pgs: 176 active+clean; 698 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3387510748' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: pool '0fa966d1-3729-426d-8271-2d6b10038686' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T05:03:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: pgmap v1214: 176 pgs: 176 active+clean; 698 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:03.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3387510748' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:03.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:03 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:03.812 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:03.820+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v771) v1 ==== 221+0+0 (secure 0 0 0) 0x7f392807ccf0 con 0x7f393c100e50 2024-03-26T05:03:03.926 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:03.934+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f393c10fbf0 con 0x7f393c100e50 2024-03-26T05:03:04.815 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.820+0000 7f3939ffb640 1 -- 172.21.15.12:0/3387510748 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v772) v1 ==== 221+0+0 (secure 0 0 0) 0x7f3928043030 con 0x7f393c100e50 2024-03-26T05:03:04.815 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_bytes = 100 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f392c06da30 msgr2=0x7f392c06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f392c06da30 0x7f392c06fef0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7f3930005d00 tx=0x7f393000b410 comp rx=0 tx=0).stop 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 msgr2=0x7f393c102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 0x7f393c102640 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f3928005ab0 tx=0x7f39280042f0 comp rx=0 tx=0).stop 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 shutdown_connections 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f393c106550 0x7f393c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f392c06da30 0x7f392c06fef0 unknown :-1 s=CLOSED pgs=3524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f393c100e50 0x7f393c102640 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 --2- 172.21.15.12:0/3387510748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f393c100490 0x7f393c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 >> 172.21.15.12:0/3387510748 conn(0x7f393c077670 msgr2=0x7f393c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 shutdown_connections 2024-03-26T05:03:04.818 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:04.824+0000 7f3942c9a640 1 -- 172.21.15.12:0/3387510748 wait complete. 2024-03-26T05:03:04.828 INFO:tasks.workunit.client.0.smithi012.stderr:+ sleep 30 2024-03-26T05:03:05.147 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[41141]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3387510748' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[29863]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3387510748' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:05.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:03:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:04 smithi186 ceph-mon[33438]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T05:03:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:04 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3387510748' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:04 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T05:03:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:04 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:05.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:06.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:05 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:03:06.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:05 smithi012 ceph-mon[29863]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T05:03:06.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:05 smithi012 ceph-mon[29863]: pgmap v1217: 176 pgs: 176 active+clean; 698 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:06.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:05 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:03:06.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:05 smithi012 ceph-mon[41141]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T05:03:06.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:05 smithi012 ceph-mon[41141]: pgmap v1217: 176 pgs: 176 active+clean; 698 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:06.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:05 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T05:03:06.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:05 smithi186 ceph-mon[33438]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T05:03:06.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:05 smithi186 ceph-mon[33438]: pgmap v1217: 176 pgs: 176 active+clean; 698 KiB data, 794 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:07.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:03:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:03:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:07.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:07 smithi186 ceph-mon[33438]: pgmap v1218: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.1 KiB/s wr, 2 op/s 2024-03-26T05:03:07.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:07 smithi012 ceph-mon[29863]: pgmap v1218: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.1 KiB/s wr, 2 op/s 2024-03-26T05:03:07.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:07 smithi012 ceph-mon[41141]: pgmap v1218: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.1 KiB/s wr, 2 op/s 2024-03-26T05:03:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:08 smithi186 ceph-mon[33438]: pool '0fa966d1-3729-426d-8271-2d6b10038686' is full (reached quota's max_bytes: 100 B) 2024-03-26T05:03:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:08 smithi186 ceph-mon[33438]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:03:08.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:08 smithi186 ceph-mon[33438]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T05:03:08.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:08 smithi012 ceph-mon[29863]: pool '0fa966d1-3729-426d-8271-2d6b10038686' is full (reached quota's max_bytes: 100 B) 2024-03-26T05:03:08.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:08 smithi012 ceph-mon[29863]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:03:08.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:08 smithi012 ceph-mon[29863]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T05:03:08.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:08 smithi012 ceph-mon[41141]: pool '0fa966d1-3729-426d-8271-2d6b10038686' is full (reached quota's max_bytes: 100 B) 2024-03-26T05:03:08.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:08 smithi012 ceph-mon[41141]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:03:08.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:08 smithi012 ceph-mon[41141]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T05:03:09.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:09 smithi186 ceph-mon[33438]: pgmap v1220: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T05:03:09.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:09 smithi012 ceph-mon[29863]: pgmap v1220: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T05:03:09.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:09 smithi012 ceph-mon[41141]: pgmap v1220: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T05:03:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:11 smithi186 ceph-mon[33438]: pgmap v1221: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 3.3 KiB/s wr, 0 op/s 2024-03-26T05:03:11.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:11 smithi012 ceph-mon[29863]: pgmap v1221: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 3.3 KiB/s wr, 0 op/s 2024-03-26T05:03:11.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:11 smithi012 ceph-mon[41141]: pgmap v1221: 176 pgs: 176 active+clean; 721 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 3.3 KiB/s wr, 0 op/s 2024-03-26T05:03:13.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:03:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:03:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:13 smithi186 ceph-mon[33438]: pgmap v1222: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.9 KiB/s wr, 1 op/s 2024-03-26T05:03:13.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:13.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:13 smithi012 ceph-mon[29863]: pgmap v1222: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.9 KiB/s wr, 1 op/s 2024-03-26T05:03:13.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:13.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:13 smithi012 ceph-mon[41141]: pgmap v1222: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.9 KiB/s wr, 1 op/s 2024-03-26T05:03:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:15 smithi186 ceph-mon[33438]: pgmap v1223: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T05:03:15.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:15 smithi012 ceph-mon[29863]: pgmap v1223: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T05:03:15.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:15 smithi012 ceph-mon[41141]: pgmap v1223: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T05:03:17.148 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:03:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:03:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:17.426 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:17 smithi012 ceph-mon[29863]: pgmap v1224: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:03:17.427 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:17 smithi012 ceph-mon[41141]: pgmap v1224: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:03:17.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:17 smithi186 ceph-mon[33438]: pgmap v1224: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:03:19.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:19 smithi186 ceph-mon[33438]: pgmap v1225: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-03-26T05:03:19.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:19 smithi012 ceph-mon[29863]: pgmap v1225: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-03-26T05:03:19.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:19 smithi012 ceph-mon[41141]: pgmap v1225: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-03-26T05:03:20.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:20.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:20 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:20.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:20.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:20 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:20.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:20.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:20 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:21.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:21 smithi012 ceph-mon[29863]: pgmap v1226: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:21.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:21 smithi012 ceph-mon[41141]: pgmap v1226: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:21.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:21 smithi186 ceph-mon[33438]: pgmap v1226: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:23.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:03:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:03:23.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:23.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:23 smithi186 ceph-mon[33438]: pgmap v1227: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:23.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:23.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:23 smithi012 ceph-mon[29863]: pgmap v1227: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:23.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:23.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:23 smithi012 ceph-mon[41141]: pgmap v1227: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:25.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:25 smithi186 ceph-mon[33438]: pgmap v1228: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:25.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:25 smithi012 ceph-mon[29863]: pgmap v1228: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:25.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:25 smithi012 ceph-mon[41141]: pgmap v1228: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:27.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:03:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:27.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:27 smithi186 ceph-mon[33438]: pgmap v1229: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:27.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:27 smithi012 ceph-mon[29863]: pgmap v1229: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:27 smithi012 ceph-mon[41141]: pgmap v1229: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:29 smithi186 ceph-mon[33438]: pgmap v1230: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:29.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:29 smithi012 ceph-mon[29863]: pgmap v1230: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:29.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:29 smithi012 ceph-mon[41141]: pgmap v1230: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:31.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:31 smithi186 ceph-mon[33438]: pgmap v1231: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:31.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:31 smithi012 ceph-mon[29863]: pgmap v1231: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:31.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:31 smithi012 ceph-mon[41141]: pgmap v1231: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:33.161 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:03:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:03:33.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:33.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:33 smithi186 ceph-mon[33438]: pgmap v1232: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:33.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:33.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:33 smithi012 ceph-mon[29863]: pgmap v1232: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:33 smithi012 ceph-mon[41141]: pgmap v1232: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:34.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:34.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:34.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ pid=149733 2024-03-26T05:03:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_bytes 0 2024-03-26T05:03:34.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put two /etc/passwd 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 -- 172.21.15.12:0/4049425853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 msgr2=0x7f195410b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 --2- 172.21.15.12:0/4049425853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f195410b600 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f1948009860 tx=0x7f194802bca0 comp rx=0 tx=0).stop 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 -- 172.21.15.12:0/4049425853 shutdown_connections 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 --2- 172.21.15.12:0/4049425853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 0x7f195410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 --2- 172.21.15.12:0/4049425853 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f195410b600 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 --2- 172.21.15.12:0/4049425853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1954102800 0x7f1954102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 -- 172.21.15.12:0/4049425853 >> 172.21.15.12:0/4049425853 conn(0x7f19540775b0 msgr2=0x7f19540779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 -- 172.21.15.12:0/4049425853 shutdown_connections 2024-03-26T05:03:34.928 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 -- 172.21.15.12:0/4049425853 wait complete. 2024-03-26T05:03:34.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 Processor -- start 2024-03-26T05:03:34.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.935+0000 7f195a49d640 1 -- start start 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f195a49d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1954102800 0x7f1954100450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f195a49d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f1954100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f195a49d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 0x7f1954100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f195a49d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19541a5550 con 0x7f195410bb40 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f195a49d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19541a56c0 con 0x7f1954102800 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f195a49d640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19541a5880 con 0x7f1954103120 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f1954100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f1954100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50626/0 (socket says 172.21.15.12:50626) 2024-03-26T05:03:34.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f1958a3a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 0x7f1954100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 -- 172.21.15.12:0/2035357628 learned_addr learned my addr 172.21.15.12:0/2035357628 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f1953fff640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1954102800 0x7f1954100450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 -- 172.21.15.12:0/2035357628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1954102800 msgr2=0x7f1954100450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1954102800 0x7f1954100450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 -- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 msgr2=0x7f1954100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 0x7f1954100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c009900 con 0x7f1954103120 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f1958a3a640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 0x7f1954100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.936+0000 7f19537fe640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f1954100990 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f1948037040 tx=0x7f19480346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.937+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948005230 con 0x7f1954103120 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.937+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1948009490 con 0x7f1954103120 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.937+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f194802cac0 con 0x7f1954103120 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.937+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480049d0 con 0x7f1954103120 2024-03-26T05:03:34.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.937+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19541a5e70 con 0x7f1954103120 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.938+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1954068470 con 0x7f1954103120 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.938+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f194803b040 con 0x7f1954103120 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.939+0000 7f19517fa640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f193c06da30 0x7f193c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.939+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(773..773 src has 259..773) v4 ==== 9212+0+0 (secure 0 0 0) 0x7f19480afe30 con 0x7f1954103120 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.939+0000 7f1953fff640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f193c06da30 0x7f193c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.939+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=774}) v3 -- 0x7f193c0797a0 con 0x7f1954103120 2024-03-26T05:03:34.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.939+0000 7f1953fff640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f193c06da30 0x7f193c06fef0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f1944005fd0 tx=0x7f1944005850 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:34.936 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:34.943+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1948078b50 con 0x7f1954103120 2024-03-26T05:03:35.120 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:35.128+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f19541a5a40 con 0x7f1954103120 2024-03-26T05:03:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:35.304+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 7 ==== osd_map(774..774 src has 259..774) v4 ==== 628+0+0 (secure 0 0 0) 0x7f194807cfb0 con 0x7f1954103120 2024-03-26T05:03:35.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:35.304+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=775}) v3 -- 0x7f193c07a2d0 con 0x7f1954103120 2024-03-26T05:03:35.302 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:35.310+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v774) v1 ==== 217+0+0 (secure 0 0 0) 0x7f1948039020 con 0x7f1954103120 2024-03-26T05:03:35.412 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:35.420+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f19541063d0 con 0x7f1954103120 2024-03-26T05:03:35.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:35 smithi012 ceph-mon[29863]: pgmap v1233: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:35.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:35 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2035357628' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:35.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:35 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:35.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:35 smithi012 ceph-mon[41141]: pgmap v1233: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:35.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:35 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2035357628' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:35.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:35 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:35.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:35 smithi186 ceph-mon[33438]: pgmap v1233: 176 pgs: 176 active+clean; 721 KiB data, 785 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:35.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:35 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2035357628' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:35.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:35 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:36.308 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.315+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 9 ==== osd_map(775..775 src has 259..775) v4 ==== 628+0+0 (secure 0 0 0) 0x7f1948080a00 con 0x7f1954103120 2024-03-26T05:03:36.308 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.315+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=776}) v3 -- 0x7f193c07a700 con 0x7f1954103120 2024-03-26T05:03:36.311 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.319+0000 7f19517fa640 1 -- 172.21.15.12:0/2035357628 <== mon.2 v2:172.21.15.12:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v775) v1 ==== 217+0+0 (secure 0 0 0) 0x7f1948039020 con 0x7f1954103120 2024-03-26T05:03:36.312 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_bytes = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f193c06da30 msgr2=0x7f193c06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f193c06da30 0x7f193c06fef0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f1944005fd0 tx=0x7f1944005850 comp rx=0 tx=0).stop 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 msgr2=0x7f1954100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f1954100990 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f1948037040 tx=0x7f19480346f0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 shutdown_connections 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f195410bb40 0x7f1954100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f193c06da30 0x7f193c06fef0 unknown :-1 s=CLOSED pgs=3526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1954103120 0x7f1954100990 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 --2- 172.21.15.12:0/2035357628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1954102800 0x7f1954100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 >> 172.21.15.12:0/2035357628 conn(0x7f19540775b0 msgr2=0x7f19541035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 shutdown_connections 2024-03-26T05:03:36.316 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.323+0000 7f195a49d640 1 -- 172.21.15.12:0/2035357628 wait complete. 2024-03-26T05:03:36.326 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_objects 0 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 -- 172.21.15.12:0/2789843870 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc41064f0 msgr2=0x7fadc41068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 --2- 172.21.15.12:0/2789843870 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc41064f0 0x7fadc41068d0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009890 tx=0x7fadb402bb70 comp rx=0 tx=0).stop 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 -- 172.21.15.12:0/2789843870 shutdown_connections 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 --2- 172.21.15.12:0/2789843870 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc4100e10 0x7fadc410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 --2- 172.21.15.12:0/2789843870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4100450 0x7fadc41008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 --2- 172.21.15.12:0/2789843870 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc41064f0 0x7fadc41068d0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 -- 172.21.15.12:0/2789843870 >> 172.21.15.12:0/2789843870 conn(0x7fadc40775b0 msgr2=0x7fadc40779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 -- 172.21.15.12:0/2789843870 shutdown_connections 2024-03-26T05:03:36.416 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 -- 172.21.15.12:0/2789843870 wait complete. 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 Processor -- start 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.421+0000 7fadc968f640 1 -- start start 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc4100450 0x7fadc41971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4100e10 0x7fadc4197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc41064f0 0x7fadc419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc419c050 con 0x7fadc41064f0 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc419c1c0 con 0x7fadc4100e10 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc419c380 con 0x7fadc4100450 2024-03-26T05:03:36.417 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc4100450 0x7fadc41971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc41064f0 0x7fadc419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc2ffd640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc4100450 0x7fadc41971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50646/0 (socket says 172.21.15.12:50646) 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc2ffd640 1 -- 172.21.15.12:0/2620672694 learned_addr learned my addr 172.21.15.12:0/2620672694 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc27fc640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4100e10 0x7fadc4197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 -- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc4100450 msgr2=0x7fadc41971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc4100450 0x7fadc41971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 -- 172.21.15.12:0/2620672694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4100e10 msgr2=0x7fadc4197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4100e10 0x7fadc4197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc009900 con 0x7fadc41064f0 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc37fe640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc41064f0 0x7fadc419ba50 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00efc0 tx=0x7fadbc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:36.418 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc003970 con 0x7fadc41064f0 2024-03-26T05:03:36.419 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fadbc004020 con 0x7fadc41064f0 2024-03-26T05:03:36.419 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb4009490 con 0x7fadc41064f0 2024-03-26T05:03:36.419 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00b870 con 0x7fadc41064f0 2024-03-26T05:03:36.419 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.422+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc41a3d00 con 0x7fadc41064f0 2024-03-26T05:03:36.419 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.423+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc4068470 con 0x7fadc41064f0 2024-03-26T05:03:36.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.424+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7fadbc00a040 con 0x7fadc41064f0 2024-03-26T05:03:36.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.424+0000 7fadabfff640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fadac06da30 0x7fadac06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:36.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.424+0000 7fadc2ffd640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fadac06da30 0x7fadac06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:36.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.424+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(775..775 src has 259..775) v4 ==== 9212+0+0 (secure 0 0 0) 0x7fadbc08e100 con 0x7fadc41064f0 2024-03-26T05:03:36.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.425+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=776}) v3 -- 0x7fadac079740 con 0x7fadc41064f0 2024-03-26T05:03:36.420 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.425+0000 7fadc2ffd640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fadac06da30 0x7fadac06fef0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fadb4005e00 tx=0x7fadb4034ab0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:36.422 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.429+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fadbc056e70 con 0x7fadc41064f0 2024-03-26T05:03:36.606 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:36.614+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fadc41089c0 con 0x7fadc41064f0 2024-03-26T05:03:36.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[29863]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2035357628' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[41141]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2035357628' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:36.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:36.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:03:36.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:36 smithi186 ceph-mon[33438]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T05:03:36.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:36 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2035357628' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:36.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:36 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:03:37.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:03:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:37.310 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.317+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v776) v1 ==== 221+0+0 (secure 0 0 0) 0x7fadbc05a530 con 0x7fadc41064f0 2024-03-26T05:03:37.310 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.317+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 8 ==== osd_map(776..776 src has 259..776) v4 ==== 628+0+0 (secure 0 0 0) 0x7fadbc05ec70 con 0x7fadc41064f0 2024-03-26T05:03:37.310 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.317+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=777}) v3 -- 0x7fadac07a620 con 0x7fadc41064f0 2024-03-26T05:03:37.426 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.434+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fadc4110d80 con 0x7fadc41064f0 2024-03-26T05:03:37.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[29863]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[29863]: pgmap v1236: 176 pgs: 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[41141]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:03:37.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:37 smithi012 ceph-mon[41141]: pgmap v1236: 176 pgs: 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:37.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:37 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:03:37.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:37 smithi186 ceph-mon[33438]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T05:03:37.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:37 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:03:37.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:37 smithi186 ceph-mon[33438]: pgmap v1236: 176 pgs: 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:37.807 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.814+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v777) v1 ==== 221+0+0 (secure 0 0 0) 0x7fadbc05a710 con 0x7fadc41064f0 2024-03-26T05:03:37.807 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_objects = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:03:37.807 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.814+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 <== mon.0 v2:172.21.15.12:3300/0 10 ==== osd_map(777..777 src has 259..777) v4 ==== 628+0+0 (secure 0 0 0) 0x7fadbc063230 con 0x7fadc41064f0 2024-03-26T05:03:37.807 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.814+0000 7fadabfff640 1 -- 172.21.15.12:0/2620672694 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=778}) v3 -- 0x7fadac07a8a0 con 0x7fadc41064f0 2024-03-26T05:03:37.810 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.817+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fadac06da30 msgr2=0x7fadac06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:37.810 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.817+0000 7fadc968f640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fadac06da30 0x7fadac06fef0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fadb4005e00 tx=0x7fadb4034ab0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.810 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.817+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc41064f0 msgr2=0x7fadc419ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:37.810 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.817+0000 7fadc968f640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc41064f0 0x7fadc419ba50 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00efc0 tx=0x7fadbc00c800 comp rx=0 tx=0).stop 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.818+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 shutdown_connections 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.818+0000 7fadc968f640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fadc41064f0 0x7fadc419ba50 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.818+0000 7fadc968f640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4100e10 0x7fadc4197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.818+0000 7fadc968f640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fadac06da30 0x7fadac06fef0 unknown :-1 s=CLOSED pgs=3527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.818+0000 7fadc968f640 1 --2- 172.21.15.12:0/2620672694 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fadc4100450 0x7fadc41971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.818+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 >> 172.21.15.12:0/2620672694 conn(0x7fadc40775b0 msgr2=0x7fadc4103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.819+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 shutdown_connections 2024-03-26T05:03:37.811 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.819+0000 7fadc968f640 1 -- 172.21.15.12:0/2620672694 wait complete. 2024-03-26T05:03:37.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ wait 149733 2024-03-26T05:03:37.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T05:03:37.827 INFO:tasks.workunit.client.0.smithi012.stderr:+ true 2024-03-26T05:03:37.827 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put three /etc/passwd 2024-03-26T05:03:37.864 INFO:tasks.workunit.client.0.smithi012.stderr:++ uuidgen 2024-03-26T05:03:37.864 INFO:tasks.workunit.client.0.smithi012.stderr:+ pp=322ac261-72fc-458a-886a-bb2cf6daea82 2024-03-26T05:03:37.864 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool create 322ac261-72fc-458a-886a-bb2cf6daea82 12 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 -- 172.21.15.12:0/4223574145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100e10 msgr2=0x7f135810d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 --2- 172.21.15.12:0/4223574145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100e10 0x7f135810d320 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f1350009c10 tx=0x7f135002bef0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 -- 172.21.15.12:0/4223574145 shutdown_connections 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 --2- 172.21.15.12:0/4223574145 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100e10 0x7f135810d320 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 --2- 172.21.15.12:0/4223574145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1358100450 0x7f13581008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 --2- 172.21.15.12:0/4223574145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13581064f0 0x7f13581068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.952 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 -- 172.21.15.12:0/4223574145 >> 172.21.15.12:0/4223574145 conn(0x7f13580775b0 msgr2=0x7f13580779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 -- 172.21.15.12:0/4223574145 shutdown_connections 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.957+0000 7f135e872640 1 -- 172.21.15.12:0/4223574145 wait complete. 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 Processor -- start 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 -- start start 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 0x7f1358196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1358100e10 0x7f13581965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13581064f0 0x7f135819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135819af50 con 0x7f13581064f0 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135819b0c0 con 0x7f1358100e10 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135e872640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135819b280 con 0x7f1358100450 2024-03-26T05:03:37.953 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f1357fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 0x7f1358196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f135ce0f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13581064f0 0x7f135819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f1357fff640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 0x7f1358196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:50700/0 (socket says 172.21.15.12:50700) 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f1357fff640 1 -- 172.21.15.12:0/3959761806 learned_addr learned my addr 172.21.15.12:0/3959761806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.958+0000 7f13577fe640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1358100e10 0x7f13581965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f1357fff640 1 -- 172.21.15.12:0/3959761806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1358100e10 msgr2=0x7f13581965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f1357fff640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1358100e10 0x7f13581965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f1357fff640 1 -- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13581064f0 msgr2=0x7f135819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f1357fff640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13581064f0 0x7f135819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f1357fff640 1 -- 172.21.15.12:0/3959761806 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340009560 con 0x7f1358100450 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f135ce0f640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13581064f0 0x7f135819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f1357fff640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 0x7f1358196090 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f134800eb00 tx=0x7f134800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:37.954 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348013070 con 0x7f1358100450 2024-03-26T05:03:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1350009840 con 0x7f1358100450 2024-03-26T05:03:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f13480054a0 con 0x7f1358100450 2024-03-26T05:03:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348004cc0 con 0x7f1358100450 2024-03-26T05:03:37.955 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.959+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13580688c0 con 0x7f1358100450 2024-03-26T05:03:37.956 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.960+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1358068470 con 0x7f1358100450 2024-03-26T05:03:37.956 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.960+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f1348004e20 con 0x7f1358100450 2024-03-26T05:03:37.956 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.961+0000 7f13557fa640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f134406da30 0x7f134406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:37.956 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.961+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(777..777 src has 259..777) v4 ==== 9212+0+0 (secure 0 0 0) 0x7f134808f630 con 0x7f1358100450 2024-03-26T05:03:37.956 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.961+0000 7f13577fe640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f134406da30 0x7f134406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:37.956 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.961+0000 7f13577fe640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f134406da30 0x7f134406fef0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f1358069f80 tx=0x7f1340009260 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:37.958 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:37.966+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1348058400 con 0x7f1358100450 2024-03-26T05:03:38.141 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.148+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12} v 0) v1 -- 0x7f135810fa80 con 0x7f1358100450 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: pool '0fa966d1-3729-426d-8271-2d6b10038686' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3959761806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:03:38.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T05:03:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:03:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: pool '0fa966d1-3729-426d-8271-2d6b10038686' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:03:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:03:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:03:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T05:03:38.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:38 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3959761806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: pool '0fa966d1-3729-426d-8271-2d6b10038686' no longer out of quota; removing NO_QUOTA flag 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2620672694' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T05:03:38.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:38 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3959761806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:38.817 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.825+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]=0 pool '322ac261-72fc-458a-886a-bb2cf6daea82' created v778) v1 ==== 176+0+0 (secure 0 0 0) 0x7f134805bac0 con 0x7f1358100450 2024-03-26T05:03:38.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.938+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12} v 0) v1 -- 0x7f135810fd30 con 0x7f1358100450 2024-03-26T05:03:38.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.939+0000 7f13557fa640 1 -- 172.21.15.12:0/3959761806 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]=0 pool '322ac261-72fc-458a-886a-bb2cf6daea82' already exists v778) v1 ==== 183+0+0 (secure 0 0 0) 0x7f134805bca0 con 0x7f1358100450 2024-03-26T05:03:38.931 INFO:tasks.workunit.client.0.smithi012.stderr:pool '322ac261-72fc-458a-886a-bb2cf6daea82' already exists 2024-03-26T05:03:38.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f134406da30 msgr2=0x7f134406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f134406da30 0x7f134406fef0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f1358069f80 tx=0x7f1340009260 comp rx=0 tx=0).stop 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 msgr2=0x7f1358196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 0x7f1358196090 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f134800eb00 tx=0x7f134800ef00 comp rx=0 tx=0).stop 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 shutdown_connections 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f13581064f0 0x7f135819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1358100e10 0x7f13581965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f134406da30 0x7f134406fef0 unknown :-1 s=CLOSED pgs=3529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 --2- 172.21.15.12:0/3959761806 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1358100450 0x7f1358196090 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 >> 172.21.15.12:0/3959761806 conn(0x7f13580775b0 msgr2=0x7f13580ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 shutdown_connections 2024-03-26T05:03:38.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:38.941+0000 7f135e872640 1 -- 172.21.15.12:0/3959761806 wait complete. 2024-03-26T05:03:38.942 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool application enable 322ac261-72fc-458a-886a-bb2cf6daea82 rados 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 -- 172.21.15.12:0/3664802255 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 msgr2=0x7f1e8c1068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 --2- 172.21.15.12:0/3664802255 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c1068d0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009930 tx=0x7f1e8402bc30 comp rx=0 tx=0).stop 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 -- 172.21.15.12:0/3664802255 shutdown_connections 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 --2- 172.21.15.12:0/3664802255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 0x7f1e8c10d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 --2- 172.21.15.12:0/3664802255 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 --2- 172.21.15.12:0/3664802255 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c1068d0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 -- 172.21.15.12:0/3664802255 >> 172.21.15.12:0/3664802255 conn(0x7f1e8c0775b0 msgr2=0x7f1e8c0779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 -- 172.21.15.12:0/3664802255 shutdown_connections 2024-03-26T05:03:39.028 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.033+0000 7f1e91a35640 1 -- 172.21.15.12:0/3664802255 wait complete. 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 Processor -- start 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 -- start start 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 0x7f1e8c1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19c060 con 0x7f1e8c100450 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19c1d0 con 0x7f1e8c100e10 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e91a35640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c19c390 con 0x7f1e8c1064f0 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e8b7fe640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56804/0 (socket says 172.21.15.12:56804) 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e8affd640 1 -- 172.21.15.12:0/2869638352 learned_addr learned my addr 172.21.15.12:0/2869638352 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:03:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.034+0000 7f1e8a7fc640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 0x7f1e8c1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8affd640 1 -- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 msgr2=0x7f1e8c19ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8affd640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c19ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8affd640 1 -- 172.21.15.12:0/2869638352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 msgr2=0x7f1e8c1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8affd640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 0x7f1e8c1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8affd640 1 -- 172.21.15.12:0/2869638352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74009490 con 0x7f1e8c100450 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8b7fe640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c19ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8a7fc640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 0x7f1e8c1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e8affd640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1971a0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009230 tx=0x7f1e840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e840044e0 con 0x7f1e8c100450 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e84009560 con 0x7f1e8c100450 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1e84003c80 con 0x7f1e8c100450 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e84005510 con 0x7f1e8c100450 2024-03-26T05:03:39.030 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.035+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c1a3d10 con 0x7f1e8c100450 2024-03-26T05:03:39.031 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.036+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c068470 con 0x7f1e8c100450 2024-03-26T05:03:39.031 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.036+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f1e84004d90 con 0x7f1e8c100450 2024-03-26T05:03:39.031 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.037+0000 7f1e90a33640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1e7806da30 0x7f1e7806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:39.032 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.037+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(778..778 src has 259..778) v4 ==== 9587+0+0 (secure 0 0 0) 0x7f1e840b04c0 con 0x7f1e8c100450 2024-03-26T05:03:39.032 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.037+0000 7f1e8a7fc640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1e7806da30 0x7f1e7806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:39.032 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.037+0000 7f1e8a7fc640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1e7806da30 0x7f1e7806fef0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c077cf0 tx=0x7f1e74005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:39.034 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.042+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1e84079060 con 0x7f1e8c100450 2024-03-26T05:03:39.218 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.225+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"} v 0) v1 -- 0x7f1e8c1089c0 con 0x7f1e8c100450 2024-03-26T05:03:39.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]': finished 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3959761806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: pgmap v1240: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]: dispatch 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]': finished 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T05:03:39.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3959761806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: pgmap v1240: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:03:39.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:39 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]: dispatch 2024-03-26T05:03:39.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]': finished 2024-03-26T05:03:39.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T05:03:39.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3959761806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pg_num": 12}]: dispatch 2024-03-26T05:03:39.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: pgmap v1240: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T05:03:39.707 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:39 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]: dispatch 2024-03-26T05:03:39.816 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.824+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]=0 enabled application 'rados' on pool '322ac261-72fc-458a-886a-bb2cf6daea82' v779) v1 ==== 213+0+0 (secure 0 0 0) 0x7f1e8407c720 con 0x7f1e8c100450 2024-03-26T05:03:39.937 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:39.945+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"} v 0) v1 -- 0x7f1e8c110d00 con 0x7f1e8c100450 2024-03-26T05:03:40.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:40 smithi186 ceph-mon[33438]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:03:40.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]': finished 2024-03-26T05:03:40.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:40 smithi186 ceph-mon[33438]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T05:03:40.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:40 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]: dispatch 2024-03-26T05:03:40.818 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.826+0000 7f1e90a33640 1 -- 172.21.15.12:0/2869638352 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]=0 enabled application 'rados' on pool '322ac261-72fc-458a-886a-bb2cf6daea82' v780) v1 ==== 213+0+0 (secure 0 0 0) 0x7f1e84043020 con 0x7f1e8c100450 2024-03-26T05:03:40.819 INFO:tasks.workunit.client.0.smithi012.stderr:enabled application 'rados' on pool '322ac261-72fc-458a-886a-bb2cf6daea82' 2024-03-26T05:03:40.822 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1e7806da30 msgr2=0x7f1e7806fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:40.822 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1e7806da30 0x7f1e7806fef0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c077cf0 tx=0x7f1e74005f90 comp rx=0 tx=0).stop 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 msgr2=0x7f1e8c1971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1971a0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009230 tx=0x7f1e840042f0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 shutdown_connections 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1e7806da30 0x7f1e7806fef0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1e8c1064f0 0x7f1e8c19ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e8c100e10 0x7f1e8c1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 --2- 172.21.15.12:0/2869638352 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e8c100450 0x7f1e8c1971a0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.830+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 >> 172.21.15.12:0/2869638352 conn(0x7f1e8c0775b0 msgr2=0x7f1e8c103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:40.824 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.831+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 shutdown_connections 2024-03-26T05:03:40.824 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.831+0000 7f1e91a35640 1 -- 172.21.15.12:0/2869638352 wait complete. 2024-03-26T05:03:40.836 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 322ac261-72fc-458a-886a-bb2cf6daea82 max_objects 10 2024-03-26T05:03:40.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[29863]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:03:40.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]': finished 2024-03-26T05:03:40.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[29863]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T05:03:40.898 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]: dispatch 2024-03-26T05:03:40.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[41141]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:03:40.898 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]': finished 2024-03-26T05:03:40.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[41141]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T05:03:40.899 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:40 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]: dispatch 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.933+0000 7f17397d7640 1 -- 172.21.15.12:0/4197628151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17341064f0 msgr2=0x7f17341068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.933+0000 7f17397d7640 1 --2- 172.21.15.12:0/4197628151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17341064f0 0x7f17341068d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f172c009930 tx=0x7f172c02bc30 comp rx=0 tx=0).stop 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 -- 172.21.15.12:0/4197628151 shutdown_connections 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 --2- 172.21.15.12:0/4197628151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1734100e10 0x7f173410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 --2- 172.21.15.12:0/4197628151 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100450 0x7f17341008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 --2- 172.21.15.12:0/4197628151 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f17341064f0 0x7f17341068d0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 -- 172.21.15.12:0/4197628151 >> 172.21.15.12:0/4197628151 conn(0x7f17340775b0 msgr2=0x7f17340779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 -- 172.21.15.12:0/4197628151 shutdown_connections 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 -- 172.21.15.12:0/4197628151 wait complete. 2024-03-26T05:03:40.929 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 Processor -- start 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.934+0000 7f17397d7640 1 -- start start 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 0x7f1734197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 0x7f17341976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17341064f0 0x7f173419b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173419bfe0 con 0x7f1734100450 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173419c150 con 0x7f17341064f0 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173419c310 con 0x7f1734100e10 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f1732ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 0x7f1734197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 0x7f17341976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 0x7f17341976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:56244/0 (socket says 172.21.15.12:56244) 2024-03-26T05:03:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f1732ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 0x7f1734197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56822/0 (socket says 172.21.15.12:56822) 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 -- 172.21.15.12:0/433919123 learned_addr learned my addr 172.21.15.12:0/433919123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17337fe640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17341064f0 0x7f173419b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 -- 172.21.15.12:0/433919123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17341064f0 msgr2=0x7f173419b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17341064f0 0x7f173419b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 -- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 msgr2=0x7f1734197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 0x7f1734197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 -- 172.21.15.12:0/433919123 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172c009560 con 0x7f1734100e10 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f1732ffd640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 0x7f1734197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17327fc640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 0x7f17341976b0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e9f0 tx=0x7f171c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c009700 con 0x7f1734100e10 2024-03-26T05:03:40.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f171c00cd80 con 0x7f1734100e10 2024-03-26T05:03:40.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173419c5f0 con 0x7f1734100e10 2024-03-26T05:03:40.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171c018760 con 0x7f1734100e10 2024-03-26T05:03:40.932 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.935+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17341a3e70 con 0x7f1734100e10 2024-03-26T05:03:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.937+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1734068470 con 0x7f1734100e10 2024-03-26T05:03:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.937+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f171c010430 con 0x7f1734100e10 2024-03-26T05:03:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.937+0000 7f172bfff640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f172006da30 0x7f172006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:03:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.937+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(780..780 src has 259..780) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f171c014070 con 0x7f1734100e10 2024-03-26T05:03:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.937+0000 7f1732ffd640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f172006da30 0x7f172006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:03:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.938+0000 7f1732ffd640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f172006da30 0x7f172006fef0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f172c02c6b0 tx=0x7f172c005fd0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:03:40.934 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:40.942+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f171c053280 con 0x7f1734100e10 2024-03-26T05:03:41.122 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:41.129+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f1734100190 con 0x7f1734100e10 2024-03-26T05:03:41.833 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:41.841+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 322ac261-72fc-458a-886a-bb2cf6daea82 v781) v1 ==== 223+0+0 (secure 0 0 0) 0x7f171c0571e0 con 0x7f1734100e10 2024-03-26T05:03:41.945 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:41.952+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f173410fb90 con 0x7f1734100e10 2024-03-26T05:03:42.147 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]': finished 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[29863]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[29863]: pgmap v1243: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/433919123' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]': finished 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[41141]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[41141]: pgmap v1243: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/433919123' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:42.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:41 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2869638352' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "app": "rados"}]': finished 2024-03-26T05:03:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:41 smithi186 ceph-mon[33438]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T05:03:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:41 smithi186 ceph-mon[33438]: pgmap v1243: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T05:03:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:41 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/433919123' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:42.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:41 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:42.834 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.841+0000 7f172bfff640 1 -- 172.21.15.12:0/433919123 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 322ac261-72fc-458a-886a-bb2cf6daea82 v782) v1 ==== 223+0+0 (secure 0 0 0) 0x7f171c05a8a0 con 0x7f1734100e10 2024-03-26T05:03:42.834 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_objects = 10 for pool 322ac261-72fc-458a-886a-bb2cf6daea82 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f172006da30 msgr2=0x7f172006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f172006da30 0x7f172006fef0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f172c02c6b0 tx=0x7f172c005fd0 comp rx=0 tx=0).stop 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 msgr2=0x7f17341976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 0x7f17341976b0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e9f0 tx=0x7f171c00eeb0 comp rx=0 tx=0).stop 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 shutdown_connections 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17341064f0 0x7f173419b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f172006da30 0x7f172006fef0 unknown :-1 s=CLOSED pgs=3531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1734100e10 0x7f17341976b0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:42.838 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 --2- 172.21.15.12:0/433919123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1734100450 0x7f1734197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:03:42.838 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 >> 172.21.15.12:0/433919123 conn(0x7f17340775b0 msgr2=0x7f17340fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:03:42.838 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 shutdown_connections 2024-03-26T05:03:42.838 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:03:42.844+0000 7f17397d7640 1 -- 172.21.15.12:0/433919123 wait complete. 2024-03-26T05:03:42.848 INFO:tasks.workunit.client.0.smithi012.stderr:+ sleep 30 2024-03-26T05:03:43.147 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[29863]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/433919123' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[41141]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/433919123' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:43.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:42 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:43.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:03:43.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:42 smithi186 ceph-mon[33438]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T05:03:43.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:42 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/433919123' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:43.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:42 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T05:03:43.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:03:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:03:44.147 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[29863]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[29863]: pgmap v1246: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[41141]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T05:03:44.148 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:43 smithi012 ceph-mon[41141]: pgmap v1246: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T05:03:44.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:44.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:43 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "field": "max_objects", "val": "10"}]': finished 2024-03-26T05:03:44.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:43 smithi186 ceph-mon[33438]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T05:03:44.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:43 smithi186 ceph-mon[33438]: pgmap v1246: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T05:03:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:45 smithi186 ceph-mon[33438]: pgmap v1247: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 973 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T05:03:45.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:45 smithi012 ceph-mon[29863]: pgmap v1247: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 973 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T05:03:45.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:45 smithi012 ceph-mon[41141]: pgmap v1247: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 973 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T05:03:47.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:03:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:47.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:47 smithi186 ceph-mon[33438]: pgmap v1248: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-26T05:03:47.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:47 smithi012 ceph-mon[29863]: pgmap v1248: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-26T05:03:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:47 smithi012 ceph-mon[41141]: pgmap v1248: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-03-26T05:03:48.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:48 smithi186 ceph-mon[33438]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:03:48.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:48 smithi012 ceph-mon[29863]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:03:48.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:48 smithi012 ceph-mon[41141]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T05:03:49.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:49 smithi186 ceph-mon[33438]: pgmap v1249: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:03:49.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:49 smithi012 ceph-mon[29863]: pgmap v1249: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:03:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:49 smithi012 ceph-mon[41141]: pgmap v1249: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T05:03:50.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:50 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:50.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:50 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:50.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:50 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:50.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:50 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:50.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:50 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:03:50.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:50 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:03:51.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:51 smithi012 ceph-mon[29863]: pgmap v1250: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 553 B/s rd, 0 op/s 2024-03-26T05:03:51.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:51 smithi012 ceph-mon[41141]: pgmap v1250: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 553 B/s rd, 0 op/s 2024-03-26T05:03:51.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:51 smithi186 ceph-mon[33438]: pgmap v1250: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 553 B/s rd, 0 op/s 2024-03-26T05:03:53.162 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:03:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:03:53.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:53.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:53 smithi186 ceph-mon[33438]: pgmap v1251: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-26T05:03:53.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:53.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:53 smithi012 ceph-mon[29863]: pgmap v1251: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-26T05:03:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:03:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:53 smithi012 ceph-mon[41141]: pgmap v1251: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 998 B/s rd, 0 op/s 2024-03-26T05:03:55.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:55 smithi186 ceph-mon[33438]: pgmap v1252: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:55.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:55 smithi012 ceph-mon[29863]: pgmap v1252: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:55.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:55 smithi012 ceph-mon[41141]: pgmap v1252: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:57.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:03:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:03:57.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:57 smithi186 ceph-mon[33438]: pgmap v1253: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:57.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:57 smithi012 ceph-mon[29863]: pgmap v1253: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:57.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:57 smithi012 ceph-mon[41141]: pgmap v1253: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:03:59.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:03:59 smithi186 ceph-mon[33438]: pgmap v1254: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:59.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:03:59 smithi012 ceph-mon[29863]: pgmap v1254: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:03:59.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:03:59 smithi012 ceph-mon[41141]: pgmap v1254: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:01.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:00 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:04:01.397 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:00 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:04:01.397 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:00 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:04:02.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:01 smithi186 ceph-mon[33438]: pgmap v1255: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:02.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:04:02.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:01 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:04:02.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:01 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:04:02.397 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[29863]: pgmap v1255: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:02.397 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:04:02.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:04:02.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:04:02.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[41141]: pgmap v1255: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:02.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:04:02.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:04:02.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:01 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:04:03.107 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:04:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:04:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:03 smithi186 ceph-mon[33438]: pgmap v1256: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:03 smithi012 ceph-mon[29863]: pgmap v1256: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:03 smithi012 ceph-mon[41141]: pgmap v1256: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:04.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:04.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:04.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:05.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:05 smithi012 ceph-mon[29863]: pgmap v1257: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:05.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:05 smithi012 ceph-mon[41141]: pgmap v1257: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:05.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:05 smithi186 ceph-mon[33438]: pgmap v1257: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:07.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:04:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:07.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:07 smithi186 ceph-mon[33438]: pgmap v1258: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:07.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:07 smithi012 ceph-mon[29863]: pgmap v1258: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:07.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:07 smithi012 ceph-mon[41141]: pgmap v1258: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:09.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:09 smithi186 ceph-mon[33438]: pgmap v1259: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:09.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:09 smithi012 ceph-mon[29863]: pgmap v1259: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:09.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:09 smithi012 ceph-mon[41141]: pgmap v1259: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:11.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:11 smithi186 ceph-mon[33438]: pgmap v1260: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:11.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:11 smithi012 ceph-mon[29863]: pgmap v1260: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:11.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:11 smithi012 ceph-mon[41141]: pgmap v1260: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:12.850 INFO:tasks.workunit.client.0.smithi012.stderr:++ seq 1 10 2024-03-26T05:04:12.850 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:12.850 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj1 /etc/passwd 2024-03-26T05:04:12.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:12.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj2 /etc/passwd 2024-03-26T05:04:12.929 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:12.929 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj3 /etc/passwd 2024-03-26T05:04:12.966 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:12.966 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj4 /etc/passwd 2024-03-26T05:04:13.001 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:13.001 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj5 /etc/passwd 2024-03-26T05:04:13.036 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:13.036 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj6 /etc/passwd 2024-03-26T05:04:13.070 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:13.071 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj7 /etc/passwd 2024-03-26T05:04:13.104 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:13.104 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj8 /etc/passwd 2024-03-26T05:04:13.139 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:13.139 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj9 /etc/passwd 2024-03-26T05:04:13.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:04:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:04:13.179 INFO:tasks.workunit.client.0.smithi012.stderr:+ for f in `seq 1 10` 2024-03-26T05:04:13.179 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 322ac261-72fc-458a-886a-bb2cf6daea82 put obj10 /etc/passwd 2024-03-26T05:04:13.214 INFO:tasks.workunit.client.0.smithi012.stderr:+ sleep 30 2024-03-26T05:04:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:13.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:13 smithi186 ceph-mon[33438]: pgmap v1261: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:13.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:13.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:13 smithi012 ceph-mon[29863]: pgmap v1261: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:13.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:13.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:13 smithi012 ceph-mon[41141]: pgmap v1261: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:15.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:15 smithi186 ceph-mon[33438]: pgmap v1262: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:15.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:15 smithi012 ceph-mon[29863]: pgmap v1262: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:15.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:15 smithi012 ceph-mon[41141]: pgmap v1262: 188 pgs: 188 active+clean; 744 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:17.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:04:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:17.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:17 smithi186 ceph-mon[33438]: pgmap v1263: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T05:04:17.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:17 smithi012 ceph-mon[29863]: pgmap v1263: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T05:04:17.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:17 smithi012 ceph-mon[41141]: pgmap v1263: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T05:04:18.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:18 smithi186 ceph-mon[33438]: pool '322ac261-72fc-458a-886a-bb2cf6daea82' is full (reached quota's max_objects: 10) 2024-03-26T05:04:18.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:18 smithi186 ceph-mon[33438]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:04:18.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:18 smithi186 ceph-mon[33438]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T05:04:18.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:18 smithi012 ceph-mon[29863]: pool '322ac261-72fc-458a-886a-bb2cf6daea82' is full (reached quota's max_objects: 10) 2024-03-26T05:04:18.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:18 smithi012 ceph-mon[29863]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:04:18.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:18 smithi012 ceph-mon[29863]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T05:04:18.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:18 smithi012 ceph-mon[41141]: pool '322ac261-72fc-458a-886a-bb2cf6daea82' is full (reached quota's max_objects: 10) 2024-03-26T05:04:18.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:18 smithi012 ceph-mon[41141]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T05:04:18.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:18 smithi012 ceph-mon[41141]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T05:04:19.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:19 smithi186 ceph-mon[33438]: pgmap v1265: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:19.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:19 smithi012 ceph-mon[29863]: pgmap v1265: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:19.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:19 smithi012 ceph-mon[41141]: pgmap v1265: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:20.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:04:20.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:20 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:20.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:04:20.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:20 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:20.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:04:20.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:20 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:21.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:21 smithi012 ceph-mon[29863]: pgmap v1266: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:21.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:21 smithi012 ceph-mon[41141]: pgmap v1266: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:21.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:21 smithi186 ceph-mon[33438]: pgmap v1266: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:23.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:04:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:04:23.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:23 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:23.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:23 smithi186 ceph-mon[33438]: pgmap v1267: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:23.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:23 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:23.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:23 smithi012 ceph-mon[29863]: pgmap v1267: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:23.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:23 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:23.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:23 smithi012 ceph-mon[41141]: pgmap v1267: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:25.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:25 smithi186 ceph-mon[33438]: pgmap v1268: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:25.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:25 smithi012 ceph-mon[29863]: pgmap v1268: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:25.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:25 smithi012 ceph-mon[41141]: pgmap v1268: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T05:04:27.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:04:26 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:27.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:27 smithi186 ceph-mon[33438]: pgmap v1269: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:27.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:27 smithi012 ceph-mon[29863]: pgmap v1269: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:27.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:27 smithi012 ceph-mon[41141]: pgmap v1269: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:29.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:29 smithi186 ceph-mon[33438]: pgmap v1270: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-26T05:04:29.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:29 smithi012 ceph-mon[29863]: pgmap v1270: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-26T05:04:29.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:29 smithi012 ceph-mon[41141]: pgmap v1270: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-26T05:04:31.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:31 smithi186 ceph-mon[33438]: pgmap v1271: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:31.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:31 smithi012 ceph-mon[29863]: pgmap v1271: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:31.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:31 smithi012 ceph-mon[41141]: pgmap v1271: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:33.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:04:32 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:04:33.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:33 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:33.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:33 smithi186 ceph-mon[33438]: pgmap v1272: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:33.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:33 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:33.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:33 smithi012 ceph-mon[29863]: pgmap v1272: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:33 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:33.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:33 smithi012 ceph-mon[41141]: pgmap v1272: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:34.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:34 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:34.661 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:34 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:34.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:34 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:35.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:35 smithi012 ceph-mon[29863]: pgmap v1273: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:35.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:35 smithi012 ceph-mon[41141]: pgmap v1273: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:35.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:35 smithi186 ceph-mon[33438]: pgmap v1273: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:37.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:04:36 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:37.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:37 smithi186 ceph-mon[33438]: pgmap v1274: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:37.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:37 smithi012 ceph-mon[29863]: pgmap v1274: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:37.650 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:37 smithi012 ceph-mon[41141]: pgmap v1274: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:39.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:39 smithi186 ceph-mon[33438]: pgmap v1275: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:39.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:39 smithi012 ceph-mon[29863]: pgmap v1275: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:39.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:39 smithi012 ceph-mon[41141]: pgmap v1275: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:41.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:41 smithi186 ceph-mon[33438]: pgmap v1276: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:41.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:41 smithi012 ceph-mon[41141]: pgmap v1276: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:41.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:41 smithi012 ceph-mon[29863]: pgmap v1276: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:04:43.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:04:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:04:43.215 INFO:tasks.workunit.client.0.smithi012.stderr:+ rados -p 0fa966d1-3729-426d-8271-2d6b10038686 put threemore /etc/passwd 2024-03-26T05:04:43.254 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_bytes 0 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/4076475734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d8107790 msgr2=0x7fd3d8107b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/4076475734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d8107790 0x7fd3d8107b70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009930 tx=0x7fd3c802bcd0 comp rx=0 tx=0).stop 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/4076475734 shutdown_connections 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/4076475734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3d8100810 0x7fd3d810e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/4076475734 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d80ffe50 0x7fd3d81002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/4076475734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d8107790 0x7fd3d8107b70 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/4076475734 >> 172.21.15.12:0/4076475734 conn(0x7fd3d80757b0 msgr2=0x7fd3d8075bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/4076475734 shutdown_connections 2024-03-26T05:04:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.351+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/4076475734 wait complete. 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 Processor -- start 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 -- start start 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 0x7fd3d8197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3d8100810 0x7fd3d8197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d8107790 0x7fd3d819bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d819c0c0 con 0x7fd3d80ffe50 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d819c230 con 0x7fd3d8100810 2024-03-26T05:04:43.347 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3dcb6c640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d819c3f0 con 0x7fd3d8107790 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 0x7fd3d8197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 0x7fd3d8197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47456/0 (socket says 172.21.15.12:47456) 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d6d9d640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d8107790 0x7fd3d819bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 -- 172.21.15.12:0/305991943 learned_addr learned my addr 172.21.15.12:0/305991943 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3cffff640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3d8100810 0x7fd3d8197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 -- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d8107790 msgr2=0x7fd3d819bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d8107790 0x7fd3d819bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 -- 172.21.15.12:0/305991943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3d8100810 msgr2=0x7fd3d8197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.352+0000 7fd3d659c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3d8100810 0x7fd3d8197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3d659c640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c0009490 con 0x7fd3d80ffe50 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3d6d9d640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d8107790 0x7fd3d819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3d659c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 0x7fd3d8197200 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009230 tx=0x7fd3c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c80044e0 con 0x7fd3d80ffe50 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c8009560 con 0x7fd3d80ffe50 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd3c8003c80 con 0x7fd3d80ffe50 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8005510 con 0x7fd3d80ffe50 2024-03-26T05:04:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.353+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d81a3d70 con 0x7fd3d80ffe50 2024-03-26T05:04:43.350 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.354+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d80688c0 con 0x7fd3d80ffe50 2024-03-26T05:04:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.354+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7fd3c8004d90 con 0x7fd3d80ffe50 2024-03-26T05:04:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.355+0000 7fd3cf7fe640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fd3c406da30 0x7fd3c406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.355+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(783..783 src has 259..783) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fd3c80b0a20 con 0x7fd3d80ffe50 2024-03-26T05:04:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.355+0000 7fd3cffff640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fd3c406da30 0x7fd3c406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.355+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=784}) v3 -- 0x7fd3c4079f60 con 0x7fd3d80ffe50 2024-03-26T05:04:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.355+0000 7fd3cffff640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fd3c406da30 0x7fd3c406fef0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8076140 tx=0x7fd3c0005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:04:43.352 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.360+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd3c8079670 con 0x7fd3d80ffe50 2024-03-26T05:04:43.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:43 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:43.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:43 smithi186 ceph-mon[33438]: pgmap v1277: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:43.542+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fd3d8110b90 con 0x7fd3d80ffe50 2024-03-26T05:04:43.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:43 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:43.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:43 smithi012 ceph-mon[41141]: pgmap v1277: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:43.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:43 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:43.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:43 smithi012 ceph-mon[29863]: pgmap v1277: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:04:44.181 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:44.188+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v784) v1 ==== 217+0+0 (secure 0 0 0) 0x7fd3c8043070 con 0x7fd3d80ffe50 2024-03-26T05:04:44.181 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:44.189+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 8 ==== osd_map(784..784 src has 259..784) v4 ==== 628+0+0 (secure 0 0 0) 0x7fd3c8003df0 con 0x7fd3d80ffe50 2024-03-26T05:04:44.181 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:44.189+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=785}) v3 -- 0x7fd3c407a9c0 con 0x7fd3d80ffe50 2024-03-26T05:04:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:44.305+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fd3d8110da0 con 0x7fd3d80ffe50 2024-03-26T05:04:44.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:44 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:04:44.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:44 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:04:44.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:44 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:04:45.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.193+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v785) v1 ==== 217+0+0 (secure 0 0 0) 0x7fd3c807cd30 con 0x7fd3d80ffe50 2024-03-26T05:04:45.186 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_bytes = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:04:45.188 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 <== mon.0 v2:172.21.15.12:3300/0 10 ==== osd_map(785..785 src has 259..785) v4 ==== 628+0+0 (secure 0 0 0) 0x7fd3c8005670 con 0x7fd3d80ffe50 2024-03-26T05:04:45.188 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3cf7fe640 1 -- 172.21.15.12:0/305991943 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=786}) v3 -- 0x7fd3c407a950 con 0x7fd3d80ffe50 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fd3c406da30 msgr2=0x7fd3c406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fd3c406da30 0x7fd3c406fef0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8076140 tx=0x7fd3c0005f90 comp rx=0 tx=0).stop 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 msgr2=0x7fd3d8197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 0x7fd3d8197200 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009230 tx=0x7fd3c80042f0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 shutdown_connections 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7fd3c406da30 0x7fd3c406fef0 unknown :-1 s=CLOSED pgs=3543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.196+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7fd3d8107790 0x7fd3d819bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.197+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3d8100810 0x7fd3d8197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.197+0000 7fd3dcb6c640 1 --2- 172.21.15.12:0/305991943 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3d80ffe50 0x7fd3d8197200 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.190 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.197+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 >> 172.21.15.12:0/305991943 conn(0x7fd3d80757b0 msgr2=0x7fd3d8105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:04:45.190 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.197+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 shutdown_connections 2024-03-26T05:04:45.190 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.197+0000 7fd3dcb6c640 1 -- 172.21.15.12:0/305991943 wait complete. 2024-03-26T05:04:45.204 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool set-quota 0fa966d1-3729-426d-8271-2d6b10038686 max_objects 0 2024-03-26T05:04:45.295 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 -- 172.21.15.12:0/1646965613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2010bbd0 msgr2=0x7f1d2010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:45.295 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 --2- 172.21.15.12:0/1646965613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2010bbd0 0x7f1d2010e000 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009be0 tx=0x7f1d1802bf80 comp rx=0 tx=0).stop 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 -- 172.21.15.12:0/1646965613 shutdown_connections 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 --2- 172.21.15.12:0/1646965613 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d2010bbd0 0x7f1d2010e000 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 --2- 172.21.15.12:0/1646965613 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d20103120 0x7f1d2010b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 --2- 172.21.15.12:0/1646965613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20102800 0x7f1d20102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 -- 172.21.15.12:0/1646965613 >> 172.21.15.12:0/1646965613 conn(0x7f1d20077670 msgr2=0x7f1d20077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 -- 172.21.15.12:0/1646965613 shutdown_connections 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 -- 172.21.15.12:0/1646965613 wait complete. 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.301+0000 7f1d26686640 1 Processor -- start 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 -- start start 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20102800 0x7f1d20197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 0x7f1d201975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2010bbd0 0x7f1d2019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2019bef0 con 0x7f1d20103120 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2019c060 con 0x7f1d20102800 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d26686640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2019c220 con 0x7f1d2010bbd0 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 0x7f1d201975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d24c23640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2010bbd0 0x7f1d2019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d1f7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 0x7f1d201975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47472/0 (socket says 172.21.15.12:47472) 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d1f7fe640 1 -- 172.21.15.12:0/212498204 learned_addr learned my addr 172.21.15.12:0/212498204 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d1ffff640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20102800 0x7f1d20197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d24c23640 1 -- 172.21.15.12:0/212498204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20102800 msgr2=0x7f1d20197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d24c23640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20102800 0x7f1d20197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d24c23640 1 -- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 msgr2=0x7f1d201975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d24c23640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 0x7f1d201975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d24c23640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10009560 con 0x7f1d2010bbd0 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.302+0000 7f1d1f7fe640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 0x7f1d201975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:04:45.297 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.303+0000 7f1d24c23640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2010bbd0 0x7f1d2019b8f0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f1d1802c4c0 tx=0x7f1d18003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:04:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.303+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18033a80 con 0x7f1d2010bbd0 2024-03-26T05:04:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.303+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1d18033be0 con 0x7f1d2010bbd0 2024-03-26T05:04:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.303+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d18009840 con 0x7f1d2010bbd0 2024-03-26T05:04:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.303+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18034430 con 0x7f1d2010bbd0 2024-03-26T05:04:45.298 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.303+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d201a3ba0 con 0x7f1d2010bbd0 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.304+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d200688c0 con 0x7f1d2010bbd0 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.304+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f1d18004040 con 0x7f1d2010bbd0 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.305+0000 7f1d1d7fa640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1d0806da30 0x7f1d0806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.305+0000 7f1d1ffff640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1d0806da30 0x7f1d0806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.305+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 5 ==== osd_map(785..785 src has 259..785) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f1d180b0df0 con 0x7f1d2010bbd0 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.305+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=786}) v3 -- 0x7f1d08079f60 con 0x7f1d2010bbd0 2024-03-26T05:04:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.305+0000 7f1d1ffff640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1d0806da30 0x7f1d0806fef0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400ba40 tx=0x7f1d14005cc0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:04:45.302 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.310+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1d18043060 con 0x7f1d2010bbd0 2024-03-26T05:04:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:04:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:45 smithi186 ceph-mon[33438]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T05:04:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:45 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:04:45.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:45 smithi186 ceph-mon[33438]: pgmap v1279: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:45.484 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:45.492+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f1d201063d0 con 0x7f1d2010bbd0 2024-03-26T05:04:45.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:04:45.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[29863]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T05:04:45.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:04:45.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[29863]: pgmap v1279: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:45.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:04:45.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[41141]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T05:04:45.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T05:04:45.651 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:45 smithi012 ceph-mon[41141]: pgmap v1279: 188 pgs: 188 active+clean; 859 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:46.193 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:46.201+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 7 ==== osd_map(786..786 src has 259..786) v4 ==== 628+0+0 (secure 0 0 0) 0x7f1d180b4030 con 0x7f1d2010bbd0 2024-03-26T05:04:46.194 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:46.201+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=787}) v3 -- 0x7f1d0807aaf0 con 0x7f1d2010bbd0 2024-03-26T05:04:46.198 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:46.205+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v786) v1 ==== 221+0+0 (secure 0 0 0) 0x7f1d18079ab0 con 0x7f1d2010bbd0 2024-03-26T05:04:46.312 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:46.320+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f1d201063d0 con 0x7f1d2010bbd0 2024-03-26T05:04:46.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:04:46.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:46 smithi186 ceph-mon[33438]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T05:04:46.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:46 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/212498204' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:46.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:46 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:46.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:04:46.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[29863]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T05:04:46.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/212498204' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:46.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:46.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/305991943' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T05:04:46.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[41141]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T05:04:46.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/212498204' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:46.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:04:46 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:47.211 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.218+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 9 ==== osd_map(787..787 src has 259..787) v4 ==== 628+0+0 (secure 0 0 0) 0x7f1d1807df10 con 0x7f1d2010bbd0 2024-03-26T05:04:47.212 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.219+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_subscribe({osdmap=788}) v3 -- 0x7f1d0807ad20 con 0x7f1d2010bbd0 2024-03-26T05:04:47.217 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.225+0000 7f1d1d7fa640 1 -- 172.21.15.12:0/212498204 <== mon.2 v2:172.21.15.12:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 v787) v1 ==== 221+0+0 (secure 0 0 0) 0x7f1d1807d5e0 con 0x7f1d2010bbd0 2024-03-26T05:04:47.217 INFO:tasks.workunit.client.0.smithi012.stderr:set-quota max_objects = 0 for pool 0fa966d1-3729-426d-8271-2d6b10038686 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1d0806da30 msgr2=0x7f1d0806fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1d0806da30 0x7f1d0806fef0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400ba40 tx=0x7f1d14005cc0 comp rx=0 tx=0).stop 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2010bbd0 msgr2=0x7f1d2019b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2010bbd0 0x7f1d2019b8f0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f1d1802c4c0 tx=0x7f1d18003ce0 comp rx=0 tx=0).stop 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 shutdown_connections 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f1d0806da30 0x7f1d0806fef0 unknown :-1 s=CLOSED pgs=3544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f1d2010bbd0 0x7f1d2019b8f0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1d20103120 0x7f1d201975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 --2- 172.21.15.12:0/212498204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20102800 0x7f1d20197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:04:47.222 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.228+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 >> 172.21.15.12:0/212498204 conn(0x7f1d20077670 msgr2=0x7f1d201035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:04:47.222 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.229+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 shutdown_connections 2024-03-26T05:04:47.222 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:04:47.229+0000 7f1d26686640 1 -- 172.21.15.12:0/212498204 wait complete. 2024-03-26T05:04:47.232 INFO:tasks.workunit.client.0.smithi012.stderr:+ sleep 30 2024-03-26T05:04:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:04:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[29863]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T05:04:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/212498204' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[29863]: pgmap v1282: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T05:04:47.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:04:47.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[41141]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T05:04:47.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/212498204' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.649 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:47 smithi012 ceph-mon[41141]: pgmap v1282: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T05:04:47.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:04:47.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:47 smithi186 ceph-mon[33438]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T05:04:47.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:47 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/212498204' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:47 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T05:04:47.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:47 smithi186 ceph-mon[33438]: pgmap v1282: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T05:04:48.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:48 smithi012 ceph-mon[29863]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:04:48.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:48 smithi012 ceph-mon[29863]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T05:04:48.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:48 smithi012 ceph-mon[41141]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:04:48.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:48 smithi012 ceph-mon[41141]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T05:04:48.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:48 smithi186 ceph-mon[33438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "field": "max_objects", "val": "0"}]': finished 2024-03-26T05:04:48.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:48 smithi186 ceph-mon[33438]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T05:04:49.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:49 smithi012 ceph-mon[29863]: pgmap v1284: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T05:04:49.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:49 smithi012 ceph-mon[41141]: pgmap v1284: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T05:04:49.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:49 smithi186 ceph-mon[33438]: pgmap v1284: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T05:04:50.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:50 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:50.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:50 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:50.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:50 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:04:51.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:51 smithi012 ceph-mon[29863]: pgmap v1285: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T05:04:51.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:51 smithi012 ceph-mon[41141]: pgmap v1285: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T05:04:51.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:51 smithi186 ceph-mon[33438]: pgmap v1285: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T05:04:53.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:04:52 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:04:53.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:53 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:53.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:53 smithi186 ceph-mon[33438]: pgmap v1286: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T05:04:53.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:53 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:53.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:53 smithi012 ceph-mon[29863]: pgmap v1286: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T05:04:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:53 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:04:53.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:53 smithi012 ceph-mon[41141]: pgmap v1286: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T05:04:55.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:55 smithi186 ceph-mon[33438]: pgmap v1287: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T05:04:55.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:55 smithi012 ceph-mon[29863]: pgmap v1287: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T05:04:55.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:55 smithi012 ceph-mon[41141]: pgmap v1287: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T05:04:57.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:04:56 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:04:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:04:57.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:57 smithi186 ceph-mon[33438]: pgmap v1288: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:57.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:57 smithi012 ceph-mon[29863]: pgmap v1288: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:57.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:57 smithi012 ceph-mon[41141]: pgmap v1288: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:04:59.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:04:59 smithi186 ceph-mon[33438]: pgmap v1289: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-26T05:04:59.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:04:59 smithi012 ceph-mon[41141]: pgmap v1289: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-26T05:04:59.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:04:59 smithi012 ceph-mon[29863]: pgmap v1289: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-26T05:05:01.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:01 smithi186 ceph-mon[33438]: pgmap v1290: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:01.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:01 smithi012 ceph-mon[29863]: pgmap v1290: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:01.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:01 smithi012 ceph-mon[41141]: pgmap v1290: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:02.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:02 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:05:02.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:02 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:05:02.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:02 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T05:05:03.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:05:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:05:03.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:05:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:03 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:05:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:03 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:05:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:03 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:03.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:03 smithi186 ceph-mon[33438]: pgmap v1291: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:05:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:05:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:05:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:03.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[29863]: pgmap v1291: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T05:05:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T05:05:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:05:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:03.648 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:03 smithi012 ceph-mon[41141]: pgmap v1291: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:04.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:04 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:04.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:04 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:04.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:04 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:05.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:05 smithi012 ceph-mon[29863]: pgmap v1292: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:05:05.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:05 smithi012 ceph-mon[41141]: pgmap v1292: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:05:05.705 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:05 smithi186 ceph-mon[33438]: pgmap v1292: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T05:05:07.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:06 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:07.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:07 smithi186 ceph-mon[33438]: pgmap v1293: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:07.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:07 smithi012 ceph-mon[29863]: pgmap v1293: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:07.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:07 smithi012 ceph-mon[41141]: pgmap v1293: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:09.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:09 smithi186 ceph-mon[33438]: pgmap v1294: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:09.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:09 smithi012 ceph-mon[29863]: pgmap v1294: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:09.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:09 smithi012 ceph-mon[41141]: pgmap v1294: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:11.458 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:11 smithi186 ceph-mon[33438]: pgmap v1295: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:11.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:11 smithi012 ceph-mon[29863]: pgmap v1295: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:11.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:11 smithi012 ceph-mon[41141]: pgmap v1295: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:13.178 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:05:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:05:13.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:13 smithi186 ceph-mon[33438]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:13.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:13 smithi186 ceph-mon[33438]: pgmap v1296: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:13.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:13 smithi012 ceph-mon[29863]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:13.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:13 smithi012 ceph-mon[29863]: pgmap v1296: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:13.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:13 smithi012 ceph-mon[41141]: from='client.24416 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T05:05:13.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:13 smithi012 ceph-mon[41141]: pgmap v1296: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:15.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:15 smithi186 ceph-mon[33438]: pgmap v1297: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:15.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:15 smithi012 ceph-mon[29863]: pgmap v1297: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:15.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:15 smithi012 ceph-mon[41141]: pgmap v1297: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T05:05:17.147 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:16 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: ::ffff:172.21.15.186 - - [26/Mar/2024:05:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T05:05:17.233 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool delete 0fa966d1-3729-426d-8271-2d6b10038686 0fa966d1-3729-426d-8271-2d6b10038686 --yes-i-really-really-mean-it 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 -- 172.21.15.12:0/1652285855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 msgr2=0x7f2f2c10d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 --2- 172.21.15.12:0/1652285855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c10d850 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009900 tx=0x7f2f2002bcd0 comp rx=0 tx=0).stop 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 -- 172.21.15.12:0/1652285855 shutdown_connections 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 --2- 172.21.15.12:0/1652285855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c10dd90 0x7f2f2c1101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 --2- 172.21.15.12:0/1652285855 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c10d850 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 --2- 172.21.15.12:0/1652285855 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c0739c0 0x7f2f2c073da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 -- 172.21.15.12:0/1652285855 >> 172.21.15.12:0/1652285855 conn(0x7f2f2c0fd020 msgr2=0x7f2f2c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 -- 172.21.15.12:0/1652285855 shutdown_connections 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.335+0000 7f2f34058640 1 -- 172.21.15.12:0/1652285855 wait complete. 2024-03-26T05:05:17.330 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 Processor -- start 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 -- start start 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c0739c0 0x7f2f2c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c10dd90 0x7f2f2c19dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c19e290 con 0x7f2f2c0742e0 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c19e400 con 0x7f2f2c0739c0 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f34058640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c19e5c0 con 0x7f2f2c10dd90 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f325f5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c10dd90 0x7f2f2c19dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57130/0 (socket says 172.21.15.12:57130) 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 -- 172.21.15.12:0/2678119217 learned_addr learned my addr 172.21.15.12:0/2678119217 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f31df4640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c0739c0 0x7f2f2c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:17.331 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 -- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c10dd90 msgr2=0x7f2f2c19dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c10dd90 0x7f2f2c19dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 -- 172.21.15.12:0/2678119217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c0739c0 msgr2=0x7f2f2c1993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c0739c0 0x7f2f2c1993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f315f3640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f24009490 con 0x7f2f2c0742e0 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f31df4640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c0739c0 0x7f2f2c1993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.336+0000 7f2f325f5640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c10dd90 0x7f2f2c19dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.337+0000 7f2f315f3640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c199910 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f2f20006010 tx=0x7f2f200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.337+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f200044e0 con 0x7f2f2c0742e0 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.337+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2f20003c80 con 0x7f2f2c0742e0 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.337+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f20009560 con 0x7f2f2c0742e0 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.337+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f200054f0 con 0x7f2f2c0742e0 2024-03-26T05:05:17.332 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.337+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c1a5f40 con 0x7f2f2c0742e0 2024-03-26T05:05:17.333 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.338+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c075070 con 0x7f2f2c0742e0 2024-03-26T05:05:17.333 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.338+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f2f20003800 con 0x7f2f2c0742e0 2024-03-26T05:05:17.334 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.339+0000 7f2f2affd640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2f1406da30 0x7f2f1406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:17.334 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.339+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(787..787 src has 259..787) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f2f200b0b10 con 0x7f2f2c0742e0 2024-03-26T05:05:17.334 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.339+0000 7f2f31df4640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2f1406da30 0x7f2f1406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:17.334 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.339+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=788}) v3 -- 0x7f2f14079fe0 con 0x7f2f2c0742e0 2024-03-26T05:05:17.334 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.339+0000 7f2f31df4640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2f1406da30 0x7f2f1406fef0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009db0 tx=0x7f2f24005f90 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:05:17.336 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.344+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f2f20079850 con 0x7f2f2c0742e0 2024-03-26T05:05:17.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:17 smithi186 ceph-mon[33438]: pgmap v1298: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:17.519 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:17.527+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f2c19a720 con 0x7f2f2c0742e0 2024-03-26T05:05:17.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:17 smithi012 ceph-mon[29863]: pgmap v1298: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:17.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:17 smithi012 ceph-mon[41141]: pgmap v1298: 188 pgs: 188 active+clean; 871 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T05:05:18.185 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.193+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]=0 pool '0fa966d1-3729-426d-8271-2d6b10038686' removed v788) v1 ==== 248+0+0 (secure 0 0 0) 0x7f2f20043020 con 0x7f2f2c0742e0 2024-03-26T05:05:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.193+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 8 ==== osd_map(788..788 src has 259..788) v4 ==== 296+0+0 (secure 0 0 0) 0x7f2f20003ab0 con 0x7f2f2c0742e0 2024-03-26T05:05:18.186 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.194+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=789}) v3 -- 0x7f2f1407ace0 con 0x7f2f2c0742e0 2024-03-26T05:05:18.304 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.311+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f2c112df0 con 0x7f2f2c0742e0 2024-03-26T05:05:18.304 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.312+0000 7f2f2affd640 1 -- 172.21.15.12:0/2678119217 <== mon.0 v2:172.21.15.12:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]=0 pool '0fa966d1-3729-426d-8271-2d6b10038686' does not exist v788) v1 ==== 255+0+0 (secure 0 0 0) 0x7f2f20043020 con 0x7f2f2c0742e0 2024-03-26T05:05:18.304 INFO:tasks.workunit.client.0.smithi012.stderr:pool '0fa966d1-3729-426d-8271-2d6b10038686' does not exist 2024-03-26T05:05:18.306 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2f1406da30 msgr2=0x7f2f1406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2f1406da30 0x7f2f1406fef0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009db0 tx=0x7f2f24005f90 comp rx=0 tx=0).stop 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 msgr2=0x7f2f2c199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c199910 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f2f20006010 tx=0x7f2f200042f0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 shutdown_connections 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f2f1406da30 0x7f2f1406fef0 unknown :-1 s=CLOSED pgs=3545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f2f2c10dd90 0x7f2f2c19dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f2c0742e0 0x7f2f2c199910 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 --2- 172.21.15.12:0/2678119217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f2c0739c0 0x7f2f2c1993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 >> 172.21.15.12:0/2678119217 conn(0x7f2f2c0fd020 msgr2=0x7f2f2c10f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 shutdown_connections 2024-03-26T05:05:18.307 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.314+0000 7f2f34058640 1 -- 172.21.15.12:0/2678119217 wait complete. 2024-03-26T05:05:18.315 INFO:tasks.workunit.client.0.smithi012.stderr:+ ceph osd pool delete 322ac261-72fc-458a-886a-bb2cf6daea82 322ac261-72fc-458a-886a-bb2cf6daea82 --yes-i-really-really-mean-it 2024-03-26T05:05:18.402 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- 172.21.15.12:0/573433031 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c106550 msgr2=0x7f760c106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:18.402 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- 172.21.15.12:0/573433031 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c106550 0x7f760c106930 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f7608009930 tx=0x7f760802bc30 comp rx=0 tx=0).stop 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- 172.21.15.12:0/573433031 shutdown_connections 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- 172.21.15.12:0/573433031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c100e50 0x7f760c10d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- 172.21.15.12:0/573433031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100490 0x7f760c100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- 172.21.15.12:0/573433031 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c106550 0x7f760c106930 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- 172.21.15.12:0/573433031 >> 172.21.15.12:0/573433031 conn(0x7f760c077670 msgr2=0x7f760c077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- 172.21.15.12:0/573433031 shutdown_connections 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- 172.21.15.12:0/573433031 wait complete. 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 Processor -- start 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- start start 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 0x7f760c197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100e50 0x7f760c1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:18.403 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c106550 0x7f760c19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c19c030 con 0x7f760c100e50 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c19c1a0 con 0x7f760c106550 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.408+0000 7f7613eb5640 1 -- --> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c19c360 con 0x7f760c100490 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611c51640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 0x7f760c197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100e50 0x7f760c1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611c51640 1 --2- >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 0x7f760c197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3301/0 says I am v2:172.21.15.12:55406/0 (socket says 172.21.15.12:55406) 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611c51640 1 -- 172.21.15.12:0/3939281587 learned_addr learned my addr 172.21.15.12:0/3939281587 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7612452640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c106550 0x7f760c19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 -- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 msgr2=0x7f760c197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 0x7f760c197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 -- 172.21.15.12:0/3939281587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c106550 msgr2=0x7f760c19ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c106550 0x7f760c19ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc009490 con 0x7f760c100e50 2024-03-26T05:05:18.404 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611c51640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 0x7f760c197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7612452640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c106550 0x7f760c19ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7611450640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100e50 0x7f760c1976b0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00e9f0 tx=0x7f75fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00cd80 con 0x7f760c100e50 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7608009560 con 0x7f760c100e50 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f75fc0044a0 con 0x7f760c100e50 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc003e70 con 0x7f760c100e50 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.409+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c1a3d60 con 0x7f760c100e50 2024-03-26T05:05:18.405 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.410+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c0688c0 con 0x7f760c100e50 2024-03-26T05:05:18.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.411+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) v1 ==== 91455+0+0 (secure 0 0 0) 0x7f75fc004000 con 0x7f760c100e50 2024-03-26T05:05:18.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.411+0000 7f7602ffd640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f75f806da30 0x7f75f806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T05:05:18.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.411+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(788..788 src has 259..788) v4 ==== 9212+0+0 (secure 0 0 0) 0x7f75fc014070 con 0x7f760c100e50 2024-03-26T05:05:18.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.411+0000 7f7611c51640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f75f806da30 0x7f75f806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T05:05:18.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.411+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=789}) v3 -- 0x7f75f8079740 con 0x7f760c100e50 2024-03-26T05:05:18.406 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.412+0000 7f7611c51640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f75f806da30 0x7f75f806fef0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7f7608000c00 tx=0x7f76080058d0 comp rx=0 tx=0).ready entity=mgr.24464 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T05:05:18.408 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.416+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f75fc057190 con 0x7f760c100e50 2024-03-26T05:05:18.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:18 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:18.589 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:18.597+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f760c110be0 con 0x7f760c100e50 2024-03-26T05:05:18.647 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:18 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:18.647 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:18 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.198+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]=0 pool '322ac261-72fc-458a-886a-bb2cf6daea82' removed v789) v1 ==== 248+0+0 (secure 0 0 0) 0x7f75fc05a850 con 0x7f760c100e50 2024-03-26T05:05:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.199+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 8 ==== osd_map(789..789 src has 259..789) v4 ==== 296+0+0 (secure 0 0 0) 0x7f75fc05b5f0 con 0x7f760c100e50 2024-03-26T05:05:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.199+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=790}) v3 -- 0x7f75f807a380 con 0x7f760c100e50 2024-03-26T05:05:19.311 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.319+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f760c110e20 con 0x7f760c100e50 2024-03-26T05:05:19.312 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.320+0000 7f7602ffd640 1 -- 172.21.15.12:0/3939281587 <== mon.0 v2:172.21.15.12:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]=0 pool '322ac261-72fc-458a-886a-bb2cf6daea82' does not exist v789) v1 ==== 255+0+0 (secure 0 0 0) 0x7f75fc05aa30 con 0x7f760c100e50 2024-03-26T05:05:19.312 INFO:tasks.workunit.client.0.smithi012.stderr:pool '322ac261-72fc-458a-886a-bb2cf6daea82' does not exist 2024-03-26T05:05:19.314 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f75f806da30 msgr2=0x7f75f806fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:19.314 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f75f806da30 0x7f75f806fef0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7f7608000c00 tx=0x7f76080058d0 comp rx=0 tx=0).stop 2024-03-26T05:05:19.314 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100e50 msgr2=0x7f760c1976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T05:05:19.314 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100e50 0x7f760c1976b0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00e9f0 tx=0x7f75fc00eeb0 comp rx=0 tx=0).stop 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 shutdown_connections 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760c106550 0x7f760c19ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f760c100e50 0x7f760c1976b0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:6800/715138906,v1:172.21.15.12:6801/715138906] conn(0x7f75f806da30 0x7f75f806fef0 unknown :-1 s=CLOSED pgs=3546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 --2- 172.21.15.12:0/3939281587 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x7f760c100490 0x7f760c197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 >> 172.21.15.12:0/3939281587 conn(0x7f760c077670 msgr2=0x7f760c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 shutdown_connections 2024-03-26T05:05:19.315 INFO:tasks.workunit.client.0.smithi012.stderr:2024-03-26T05:05:19.322+0000 7f7613eb5640 1 -- 172.21.15.12:0/3939281587 wait complete. 2024-03-26T05:05:19.323 INFO:tasks.workunit.client.0.smithi012.stdout:OK 2024-03-26T05:05:19.325 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo OK 2024-03-26T05:05:19.325 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T05:05:19.325 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T05:05:19.394 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-26T05:05:19.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T05:05:19.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:05:19.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:19 smithi186 ceph-mon[33438]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T05:05:19.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:19 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:19 smithi186 ceph-mon[33438]: pgmap v1300: 176 pgs: 176 active+clean; 698 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:05:19.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:05:19.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[29863]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T05:05:19.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.458 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[29863]: pgmap v1300: 176 pgs: 176 active+clean; 698 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:05:19.459 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:05:19.459 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[41141]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T05:05:19.459 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/2678119217' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0fa966d1-3729-426d-8271-2d6b10038686", "pool2": "0fa966d1-3729-426d-8271-2d6b10038686", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.459 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:19.459 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:19 smithi012 ceph-mon[41141]: pgmap v1300: 176 pgs: 176 active+clean; 698 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T05:05:19.729 DEBUG:teuthology.parallel:result is None 2024-03-26T05:05:19.730 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T05:05:19.797 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T05:05:19.797 DEBUG:teuthology.orchestra.run.smithi012:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T05:05:20.019 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T05:05:20.019 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T05:05:20.030 INFO:tasks.cephadm:Teardown begin 2024-03-26T05:05:20.030 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:05:20.058 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:05:20.094 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T05:05:20.094 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:05:20.112 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T05:05:20.152 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T05:05:20.152 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T05:05:20.152 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a 2024-03-26T05:05:20.397 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 systemd[1]: Stopping Ceph mon.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[29863]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[29863]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[29863]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[29863]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[29863]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[41141]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:05:20.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:05:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[41141]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T05:05:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[41141]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[41141]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:05:20.399 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-mon[41141]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:20.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:20 smithi186 ceph-mon[33438]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T05:05:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T05:05:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:20 smithi186 ceph-mon[33438]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T05:05:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:20 smithi186 ceph-mon[33438]: from='client.? 172.21.15.12:0/3939281587' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "322ac261-72fc-458a-886a-bb2cf6daea82", "pool2": "322ac261-72fc-458a-886a-bb2cf6daea82", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T05:05:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:20 smithi186 ceph-mon[33438]: from='mgr.24464 ' entity='mgr.y' 2024-03-26T05:05:20.456 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:20 smithi186 ceph-mon[33438]: from='mgr.24464 172.21.15.12:0/1491925607' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T05:05:20.897 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a[29859]: 2024-03-26T05:05:20.438+0000 7fe5f5538700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:20.897 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a[29859]: 2024-03-26T05:05:20.438+0000 7fe5f5538700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T05:05:20.897 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:20 smithi012 podman[150617]: 2024-03-26 05:05:20.653424236 +0000 UTC m=+0.307908512 container died 86745ba2dacf6b5669dda0190eabc3655e67d8b2ecaf37a9d3305bd4c15761ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T05:05:21.474 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:21 smithi012 podman[150617]: 2024-03-26 05:05:21.214752531 +0000 UTC m=+0.869236847 container cleanup 86745ba2dacf6b5669dda0190eabc3655e67d8b2ecaf37a9d3305bd4c15761ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T05:05:21.474 INFO:journalctl@ceph.mon.a.smithi012.stdout:Mar 26 05:05:21 smithi012 bash[150617]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-a 2024-03-26T05:05:21.637 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T05:05:21.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:21.677 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T05:05:21.678 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T05:05:21.678 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.c 2024-03-26T05:05:22.002 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:21 smithi012 systemd[1]: Stopping Ceph mon.c for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:22.397 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:05:05:22] ENGINE Bus STOPPING 2024-03-26T05:05:22.397 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:05:05:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:05:05:22] ENGINE Bus STOPPED 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:05:05:22] ENGINE Bus STARTING 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:05:05:22] ENGINE Serving on http://:::9283 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: [26/Mar/2024:05:05:22] ENGINE Bus STARTED 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-c[41137]: 2024-03-26T05:05:22.010+0000 7f3ff8669700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:22 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-c[41137]: 2024-03-26T05:05:22.010+0000 7f3ff8669700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T05:05:22.398 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:22 smithi012 podman[150741]: 2024-03-26 05:05:22.24539028 +0000 UTC m=+0.336986262 container died 57a13f14d39a2b7310b4a8c781652b317684b686b036c0b93b08c4c1a775521f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T05:05:22.897 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:22 smithi012 podman[150741]: 2024-03-26 05:05:22.752412808 +0000 UTC m=+0.844008798 container cleanup 57a13f14d39a2b7310b4a8c781652b317684b686b036c0b93b08c4c1a775521f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T05:05:22.897 INFO:journalctl@ceph.mon.c.smithi012.stdout:Mar 26 05:05:22 smithi012 bash[150741]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-c 2024-03-26T05:05:23.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T05:05:23.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:23.161 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T05:05:23.162 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T05:05:23.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.b 2024-03-26T05:05:23.191 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:05:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug there is no tcmu-runner data available 2024-03-26T05:05:23.455 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:23 smithi186 systemd[1]: Stopping Ceph mon.b for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:23.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b[33434]: 2024-03-26T05:05:23.489+0000 7fa0dac66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:23.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b[33434]: 2024-03-26T05:05:23.489+0000 7fa0dac66700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T05:05:23.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:23 smithi186 podman[69801]: 2024-03-26 05:05:23.592932479 +0000 UTC m=+0.232006545 container died 8fc10543afb1e297b001b7a48272a011dcb7dc8f640043cb7650a52a639fcdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T05:05:24.205 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:24 smithi186 podman[69801]: 2024-03-26 05:05:24.091411221 +0000 UTC m=+0.730485303 container cleanup 8fc10543afb1e297b001b7a48272a011dcb7dc8f640043cb7650a52a639fcdca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T05:05:24.206 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 26 05:05:24 smithi186 bash[69801]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mon-b 2024-03-26T05:05:24.447 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T05:05:24.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:24.533 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T05:05:24.534 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T05:05:24.534 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y 2024-03-26T05:05:24.835 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:24 smithi012 systemd[1]: Stopping Ceph mgr.y for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:24.835 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:24 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y[30094]: 2024-03-26T05:05:24.814+0000 7f7973ad8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:25.124 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:24 smithi012 podman[150876]: 2024-03-26 05:05:24.846323752 +0000 UTC m=+0.113848236 container died d473c37e3fccca5790f4966958c04a24bd12ebdf73abf8a544dc24cabb642a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T05:05:25.385 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:25 smithi012 podman[150876]: 2024-03-26 05:05:25.373865669 +0000 UTC m=+0.641390165 container cleanup d473c37e3fccca5790f4966958c04a24bd12ebdf73abf8a544dc24cabb642a60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T05:05:25.385 INFO:journalctl@ceph.mgr.y.smithi012.stdout:Mar 26 05:05:25 smithi012 bash[150876]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-y 2024-03-26T05:05:25.795 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T05:05:25.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:25.832 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T05:05:25.832 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T05:05:25.832 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.x 2024-03-26T05:05:26.153 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 05:05:25 smithi186 systemd[1]: Stopping Ceph mgr.x for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:26.455 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 05:05:26 smithi186 podman[69923]: 2024-03-26 05:05:26.202283595 +0000 UTC m=+0.158602832 container died 06682ec9514dff436e13a6a7bfa83f396932823a04f469336fd5a375e6162b19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T05:05:26.455 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:05:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:05:26.162Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:05:26.456 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:05:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:05:26.164Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:05:26.456 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:05:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:05:26.164Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:05:26.456 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:05:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:05:26.165Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:05:26.705 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 05:05:26 smithi186 podman[69923]: 2024-03-26 05:05:26.546291091 +0000 UTC m=+0.502610317 container cleanup 06682ec9514dff436e13a6a7bfa83f396932823a04f469336fd5a375e6162b19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T05:05:26.705 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 26 05:05:26 smithi186 bash[69923]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-mgr-x 2024-03-26T05:05:27.119 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T05:05:27.158 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:27.158 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T05:05:27.158 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T05:05:27.158 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.0 2024-03-26T05:05:27.469 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:27 smithi012 systemd[1]: Stopping Ceph osd.0 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:27.897 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:27 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0[45181]: 2024-03-26T05:05:27.476+0000 7fb6836b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:27.897 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:27 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0[45181]: 2024-03-26T05:05:27.476+0000 7fb6836b2700 -1 osd.0 789 *** Got signal Terminated *** 2024-03-26T05:05:27.898 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:27 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0[45181]: 2024-03-26T05:05:27.476+0000 7fb6836b2700 -1 osd.0 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:05:33.246 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:33 smithi012 podman[150997]: 2024-03-26 05:05:33.002278489 +0000 UTC m=+5.637628278 container died 2a951b5f2b0d778c375dfda88dac7243870724221ec0f3c5b5dc306622eb3493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T05:05:33.503 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:33 smithi012 podman[150997]: 2024-03-26 05:05:33.326918101 +0000 UTC m=+5.962267937 container cleanup 2a951b5f2b0d778c375dfda88dac7243870724221ec0f3c5b5dc306622eb3493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True) 2024-03-26T05:05:33.503 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:33 smithi012 bash[150997]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0 2024-03-26T05:05:33.897 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:33 smithi012 podman[151013]: 2024-03-26 05:05:33.511698377 +0000 UTC m=+0.504732752 container remove 2a951b5f2b0d778c375dfda88dac7243870724221ec0f3c5b5dc306622eb3493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T05:05:33.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:33 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:33.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:05:33.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:33 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:33.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:05:34.232 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:33 smithi012 podman[151108]: 2024-03-26 05:05:33.92915075 +0000 UTC m=+0.108121196 container create 86b021028dc336045e99b6fdd992ff2d8737f3f5dd372c23a1a2d12578d239b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:05:34.232 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:33 smithi012 podman[151108]: 2024-03-26 05:05:33.838028579 +0000 UTC m=+0.016999024 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:05:34.232 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:34 smithi012 podman[151108]: 2024-03-26 05:05:34.131393103 +0000 UTC m=+0.310363550 container init 86b021028dc336045e99b6fdd992ff2d8737f3f5dd372c23a1a2d12578d239b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T05:05:34.232 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:34 smithi012 podman[151108]: 2024-03-26 05:05:34.134536556 +0000 UTC m=+0.313507001 container start 86b021028dc336045e99b6fdd992ff2d8737f3f5dd372c23a1a2d12578d239b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T05:05:34.232 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:34 smithi012 podman[151108]: 2024-03-26 05:05:34.239835369 +0000 UTC m=+0.418805816 container attach 86b021028dc336045e99b6fdd992ff2d8737f3f5dd372c23a1a2d12578d239b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T05:05:34.647 INFO:journalctl@ceph.osd.0.smithi012.stdout:Mar 26 05:05:34 smithi012 podman[151108]: 2024-03-26 05:05:34.440693642 +0000 UTC m=+0.619664087 container died 86b021028dc336045e99b6fdd992ff2d8737f3f5dd372c23a1a2d12578d239b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124) 2024-03-26T05:05:35.041 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T05:05:35.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:35.086 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T05:05:35.086 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T05:05:35.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.1 2024-03-26T05:05:35.311 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:35 smithi012 systemd[1]: Stopping Ceph osd.1 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:35.647 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:35 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T05:05:35.438+0000 7fbe302a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:35.647 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:35 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T05:05:35.438+0000 7fbe302a1700 -1 osd.1 789 *** Got signal Terminated *** 2024-03-26T05:05:35.647 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:35 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1[51144]: 2024-03-26T05:05:35.438+0000 7fbe302a1700 -1 osd.1 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:05:41.272 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:40 smithi012 podman[151216]: 2024-03-26 05:05:40.975935515 +0000 UTC m=+5.656740981 container died 9ee46790fd56fb38f06d998c805cec5f16575587d1c9d8e978562d2154dc1cb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T05:05:41.522 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:41 smithi012 podman[151216]: 2024-03-26 05:05:41.363454153 +0000 UTC m=+6.044259576 container cleanup 9ee46790fd56fb38f06d998c805cec5f16575587d1c9d8e978562d2154dc1cb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T05:05:41.523 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:41 smithi012 bash[151216]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1 2024-03-26T05:05:41.897 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:41 smithi012 podman[151246]: 2024-03-26 05:05:41.531280778 +0000 UTC m=+0.546428750 container remove 9ee46790fd56fb38f06d998c805cec5f16575587d1c9d8e978562d2154dc1cb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T05:05:42.210 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:41 smithi012 podman[151338]: 2024-03-26 05:05:41.83689307 +0000 UTC m=+0.017468220 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:05:42.210 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:41 smithi012 podman[151338]: 2024-03-26 05:05:41.965715689 +0000 UTC m=+0.146290879 container create b2affb171b7a0f7a958cc7a660296040884b477b720e060ff5f178ad673117df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T05:05:42.508 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:42 smithi012 podman[151338]: 2024-03-26 05:05:42.218649611 +0000 UTC m=+0.399224769 container init b2affb171b7a0f7a958cc7a660296040884b477b720e060ff5f178ad673117df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T05:05:42.509 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:42 smithi012 podman[151338]: 2024-03-26 05:05:42.221934936 +0000 UTC m=+0.402510083 container start b2affb171b7a0f7a958cc7a660296040884b477b720e060ff5f178ad673117df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:05:42.509 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:42 smithi012 podman[151338]: 2024-03-26 05:05:42.301617863 +0000 UTC m=+0.482193013 container attach b2affb171b7a0f7a958cc7a660296040884b477b720e060ff5f178ad673117df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T05:05:42.796 INFO:journalctl@ceph.osd.1.smithi012.stdout:Mar 26 05:05:42 smithi012 podman[151338]: 2024-03-26 05:05:42.517450472 +0000 UTC m=+0.698025619 container died b2affb171b7a0f7a958cc7a660296040884b477b720e060ff5f178ad673117df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T05:05:43.101 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T05:05:43.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:43.147 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T05:05:43.147 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T05:05:43.147 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.2 2024-03-26T05:05:43.397 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:43 smithi012 systemd[1]: Stopping Ceph osd.2 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:43.897 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2[57219]: 2024-03-26T05:05:43.440+0000 7fa9d952a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:43.897 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2[57219]: 2024-03-26T05:05:43.440+0000 7fa9d952a700 -1 osd.2 789 *** Got signal Terminated *** 2024-03-26T05:05:43.897 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2[57219]: 2024-03-26T05:05:43.440+0000 7fa9d952a700 -1 osd.2 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:05:43.898 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:43.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:05:43.898 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:43.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:05:43.898 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:43.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:05:49.147 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:48 smithi012 podman[151443]: 2024-03-26 05:05:48.898527895 +0000 UTC m=+5.556961829 container died c547b39e68fed3881ac1f0d350ce0b7a62a4f73a4cb26de8e6fb9fd431b55380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:05:49.482 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:49 smithi012 podman[151443]: 2024-03-26 05:05:49.282034402 +0000 UTC m=+5.940468324 container cleanup c547b39e68fed3881ac1f0d350ce0b7a62a4f73a4cb26de8e6fb9fd431b55380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T05:05:49.483 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:49 smithi012 bash[151443]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2 2024-03-26T05:05:49.802 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:49 smithi012 podman[151460]: 2024-03-26 05:05:49.49165422 +0000 UTC m=+0.583145933 container remove c547b39e68fed3881ac1f0d350ce0b7a62a4f73a4cb26de8e6fb9fd431b55380 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:05:50.147 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:49 smithi012 podman[151552]: 2024-03-26 05:05:49.714034939 +0000 UTC m=+0.018202367 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:05:50.147 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:49 smithi012 podman[151552]: 2024-03-26 05:05:49.83428376 +0000 UTC m=+0.138451231 container create c57543658014029f77adf8df68625ee0db92c1b2affc368d231263bf0cea079a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T05:05:50.147 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:50 smithi012 podman[151552]: 2024-03-26 05:05:50.044757252 +0000 UTC m=+0.348924678 container init c57543658014029f77adf8df68625ee0db92c1b2affc368d231263bf0cea079a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T05:05:50.147 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:50 smithi012 podman[151552]: 2024-03-26 05:05:50.047716991 +0000 UTC m=+0.351884419 container start c57543658014029f77adf8df68625ee0db92c1b2affc368d231263bf0cea079a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:05:50.647 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:50 smithi012 podman[151552]: 2024-03-26 05:05:50.308734273 +0000 UTC m=+0.612901770 container attach c57543658014029f77adf8df68625ee0db92c1b2affc368d231263bf0cea079a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-26T05:05:50.647 INFO:journalctl@ceph.osd.2.smithi012.stdout:Mar 26 05:05:50 smithi012 podman[151552]: 2024-03-26 05:05:50.37718826 +0000 UTC m=+0.681355690 container died c57543658014029f77adf8df68625ee0db92c1b2affc368d231263bf0cea079a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T05:05:50.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T05:05:50.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:50.987 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T05:05:50.987 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T05:05:50.987 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.3 2024-03-26T05:05:51.292 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:51 smithi012 systemd[1]: Stopping Ceph osd.3 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:51.647 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:51 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3[63439]: 2024-03-26T05:05:51.300+0000 7f77c3b05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:51.647 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:51 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3[63439]: 2024-03-26T05:05:51.300+0000 7f77c3b05700 -1 osd.3 789 *** Got signal Terminated *** 2024-03-26T05:05:51.647 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:51 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3[63439]: 2024-03-26T05:05:51.300+0000 7f77c3b05700 -1 osd.3 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:05:53.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:53.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:05:53.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:53.574Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:05:53.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:05:53 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:05:53.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:05:54.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:05:54 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:05:54.534+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:05:55.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:05:55 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:05:55.492+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:05:55.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:05:55 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:05:55.483+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:05:56.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:05:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:05:56.464+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:05:56.706 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:05:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:05:56.376+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:05:56.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:05:56 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:05:56.480+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:05:56.897 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:56 smithi012 podman[151658]: 2024-03-26 05:05:56.627256246 +0000 UTC m=+5.440020427 container died 45d494c910d85d1ea276e6a1622708fc8a2b72d7f3cbb4d0b16b643b59ed6cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T05:05:57.267 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:56 smithi012 podman[151658]: 2024-03-26 05:05:56.991909711 +0000 UTC m=+5.804673890 container cleanup 45d494c910d85d1ea276e6a1622708fc8a2b72d7f3cbb4d0b16b643b59ed6cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T05:05:57.267 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:56 smithi012 bash[151658]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3 2024-03-26T05:05:57.604 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:57 smithi012 podman[151685]: 2024-03-26 05:05:57.276260211 +0000 UTC m=+0.644153476 container remove 45d494c910d85d1ea276e6a1622708fc8a2b72d7f3cbb4d0b16b643b59ed6cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T05:05:57.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:05:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:05:57.505+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:05:57.706 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:05:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:05:57.397+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:05:57.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:05:57 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:05:57.445+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:05:57.897 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:57 smithi012 podman[151779]: 2024-03-26 05:05:57.515856623 +0000 UTC m=+0.017767300 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:05:57.898 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:57 smithi012 podman[151779]: 2024-03-26 05:05:57.635455701 +0000 UTC m=+0.137366408 container create 0781f72f5e30827b4954fa49390a0b9a7aeb749bac26926a5a5517523768d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T05:05:57.898 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:57 smithi012 podman[151779]: 2024-03-26 05:05:57.845735218 +0000 UTC m=+0.347645886 container init 0781f72f5e30827b4954fa49390a0b9a7aeb749bac26926a5a5517523768d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T05:05:57.898 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:57 smithi012 podman[151779]: 2024-03-26 05:05:57.848664833 +0000 UTC m=+0.350575504 container start 0781f72f5e30827b4954fa49390a0b9a7aeb749bac26926a5a5517523768d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T05:05:58.397 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:57 smithi012 podman[151779]: 2024-03-26 05:05:57.937352964 +0000 UTC m=+0.439263630 container attach 0781f72f5e30827b4954fa49390a0b9a7aeb749bac26926a5a5517523768d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:05:58.397 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:58 smithi012 podman[151779]: 2024-03-26 05:05:58.158357026 +0000 UTC m=+0.660267693 container died 0781f72f5e30827b4954fa49390a0b9a7aeb749bac26926a5a5517523768d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T05:05:58.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T05:05:58.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:05:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:05:58.518+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:05:58.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:05:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:05:58.494+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:05:58.706 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:05:58 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:05:58.356+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:05:58.711 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:58 smithi012 podman[151805]: 2024-03-26 05:05:58.674219931 +0000 UTC m=+0.511312710 container remove 0781f72f5e30827b4954fa49390a0b9a7aeb749bac26926a5a5517523768d4b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T05:05:58.711 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:58 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T05:05:58.711 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:58 smithi012 systemd[1]: Stopped Ceph osd.3 for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T05:05:58.711 INFO:journalctl@ceph.osd.3.smithi012.stdout:Mar 26 05:05:58 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.3.service: Consumed 43.190s CPU time. 2024-03-26T05:05:58.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:05:58.727 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T05:05:58.728 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T05:05:58.728 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.4 2024-03-26T05:05:59.074 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:05:58 smithi186 systemd[1]: Stopping Ceph osd.4 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:05:59.335 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:05:59.081+0000 7fe46c782700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:05:59.335 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:05:59.081+0000 7fe46c782700 -1 osd.4 789 *** Got signal Terminated *** 2024-03-26T05:05:59.335 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:05:59.081+0000 7fe46c782700 -1 osd.4 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:05:59.705 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:05:59.343+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:05:59.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:05:59.482+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:05:59.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:05:59.494+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:05:59.706 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:05:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:05:59.467+0000 7fe468d76700 -1 osd.4 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:31.071779+0000 front 2024-03-26T05:05:31.071882+0000 (oldest deadline 2024-03-26T05:05:58.672308+0000) 2024-03-26T05:06:00.706 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:00 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:06:00.438+0000 7fe468d76700 -1 osd.4 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:31.071779+0000 front 2024-03-26T05:05:31.071882+0000 (oldest deadline 2024-03-26T05:05:58.672308+0000) 2024-03-26T05:06:00.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:00 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:00.477+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:00.706 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:00 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:00.323+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:00.707 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:00 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:00.487+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:01.705 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:06:01.419+0000 7fe468d76700 -1 osd.4 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:31.071779+0000 front 2024-03-26T05:05:31.071882+0000 (oldest deadline 2024-03-26T05:05:58.672308+0000) 2024-03-26T05:06:01.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:01.532+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:01.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:01.504+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:01.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:01.504+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:01.706 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:01 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:01.296+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:02.705 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:06:02.384+0000 7fe468d76700 -1 osd.4 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:31.071779+0000 front 2024-03-26T05:05:31.071882+0000 (oldest deadline 2024-03-26T05:05:58.672308+0000) 2024-03-26T05:06:02.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:02.495+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:02.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:02.470+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:02.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:02.470+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:02.707 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:02.250+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:02.707 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:02 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:02.250+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:03.522 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:03 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4[38369]: 2024-03-26T05:06:03.344+0000 7fe468d76700 -1 osd.4 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:31.071779+0000 front 2024-03-26T05:05:31.071882+0000 (oldest deadline 2024-03-26T05:05:58.672308+0000) 2024-03-26T05:06:03.523 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:03 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:03.469+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:03.523 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:03 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:03.469+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:03.523 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:03 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:03.226+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:03.523 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:03 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:03.226+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:03.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:03 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:03.574Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:03.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:03 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:03.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:06:03.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:03 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:03.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:03.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:03 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:03.530+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:04.439 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:04 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:04.186+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:04.439 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:04 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:04.186+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:04.705 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:04 smithi186 podman[70063]: 2024-03-26 05:06:04.536501215 +0000 UTC m=+5.598486880 container died 342836ad7ea48f737e6a176b11bee68017875ffef7cdbb80a7c6efb39012013d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:06:04.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:04 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:04.512+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:04.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:04 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:04.512+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:04.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:04 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:04.447+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:04.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:04 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:04.447+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:05.149 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:04 smithi186 podman[70063]: 2024-03-26 05:06:04.907180336 +0000 UTC m=+5.969165988 container cleanup 342836ad7ea48f737e6a176b11bee68017875ffef7cdbb80a7c6efb39012013d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T05:06:05.150 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:04 smithi186 bash[70063]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4 2024-03-26T05:06:05.455 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:05 smithi186 podman[70082]: 2024-03-26 05:06:05.158509032 +0000 UTC m=+0.619688410 container remove 342836ad7ea48f737e6a176b11bee68017875ffef7cdbb80a7c6efb39012013d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-26T05:06:05.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:05.197+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:05.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:05.197+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:05.803 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:05.520+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:05.804 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:05.520+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:05.804 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:05.472+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:05.804 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:05 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:05.472+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:05.804 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:05 smithi186 podman[70176]: 2024-03-26 05:06:05.379884111 +0000 UTC m=+0.018010947 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:06:05.805 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:05 smithi186 podman[70176]: 2024-03-26 05:06:05.501332954 +0000 UTC m=+0.139459789 container create 6a6e851fe81f01c2c35413d5fe52a9a0b912c7db200c1e92bbb2009bf92563cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T05:06:05.805 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:05 smithi186 podman[70176]: 2024-03-26 05:06:05.812665203 +0000 UTC m=+0.450792039 container init 6a6e851fe81f01c2c35413d5fe52a9a0b912c7db200c1e92bbb2009bf92563cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T05:06:06.092 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:05 smithi186 podman[70176]: 2024-03-26 05:06:05.815946336 +0000 UTC m=+0.454073172 container start 6a6e851fe81f01c2c35413d5fe52a9a0b912c7db200c1e92bbb2009bf92563cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T05:06:06.092 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:05 smithi186 podman[70176]: 2024-03-26 05:06:05.895802095 +0000 UTC m=+0.533928939 container attach 6a6e851fe81f01c2c35413d5fe52a9a0b912c7db200c1e92bbb2009bf92563cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T05:06:06.364 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 26 05:06:06 smithi186 podman[70176]: 2024-03-26 05:06:06.101201535 +0000 UTC m=+0.739328377 container died 6a6e851fe81f01c2c35413d5fe52a9a0b912c7db200c1e92bbb2009bf92563cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:06:06.364 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:06 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:06.177+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:06.364 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:06 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:06.177+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:06.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:06 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:06.526+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:06.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:06 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:06.526+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:06.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:06 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:06.499+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:06.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:06 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:06.499+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:06.760 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T05:06:06.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:06:06.801 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T05:06:06.802 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T05:06:06.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.5 2024-03-26T05:06:07.144 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:06 smithi186 systemd[1]: Stopping Ceph osd.5 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:07.455 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:07.152+0000 7f907a7c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:06:07.455 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:07.152+0000 7f907a7c2700 -1 osd.5 789 *** Got signal Terminated *** 2024-03-26T05:06:07.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:07.152+0000 7f907a7c2700 -1 osd.5 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:06:07.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:07.189+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:07.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:07.189+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:07.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:07.495+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:07.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:07.495+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:07.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:07.468+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:07.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:07 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:07.468+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:08.425 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:08 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:08.168+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:08.425 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:08 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:08.168+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:08.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:08 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:08.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:08.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:08 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:08.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:08.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:08 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:08.530+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:08.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:08 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:08.530+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:09.397 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:09.147+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:09.397 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:09.147+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:09.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:09.564+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:09.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:09.564+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:09.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:09.564+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:09.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:09.405+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:09.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:09 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:09.405+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:10.407 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:10.162+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:10.407 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:10.162+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:10.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:10.518+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:10.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:10.518+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:10.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:10.518+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:10.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:10.414+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:10.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:10 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:10.414+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:11.455 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:11.199+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.831359+0000 front 2024-03-26T05:05:29.831288+0000 (oldest deadline 2024-03-26T05:05:55.731223+0000) 2024-03-26T05:06:11.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:11.199+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.032094+0000 front 2024-03-26T05:05:38.032017+0000 (oldest deadline 2024-03-26T05:06:01.532109+0000) 2024-03-26T05:06:11.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:11.199+0000 7f9072dae700 -1 osd.5 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:47.433138+0000 front 2024-03-26T05:05:47.433118+0000 (oldest deadline 2024-03-26T05:06:10.833378+0000) 2024-03-26T05:06:11.456 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5[44434]: 2024-03-26T05:06:11.199+0000 7f9072dae700 -1 osd.5 789 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14436.0:4608 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94610832647168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e789) 2024-03-26T05:06:11.456 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:11.389+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:11.456 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:11.389+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:11.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:11.469+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:11.956 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:11.469+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:11.956 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:11 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:11.469+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:12.705 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:12 smithi186 podman[70292]: 2024-03-26 05:06:12.542192939 +0000 UTC m=+5.495293130 container died 5995b2b80e169090e08f582fe3d0671eac7e73ff452f7080fdaeb2ab3f359334 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-26T05:06:12.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:12.483+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:12.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:12.483+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:12.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:12.483+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:12.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:12.425+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:12.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:12.425+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:12.707 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:12 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:12.425+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:13.205 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:12 smithi186 podman[70292]: 2024-03-26 05:06:12.910736629 +0000 UTC m=+5.863836812 container cleanup 5995b2b80e169090e08f582fe3d0671eac7e73ff452f7080fdaeb2ab3f359334 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T05:06:13.205 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:12 smithi186 bash[70292]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5 2024-03-26T05:06:13.574 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:13 smithi186 podman[70310]: 2024-03-26 05:06:13.237061076 +0000 UTC m=+0.690884532 container remove 5995b2b80e169090e08f582fe3d0671eac7e73ff452f7080fdaeb2ab3f359334 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T05:06:13.575 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:13.474+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:13.575 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:13.474+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:13.575 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:13.474+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:13.575 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:13.440+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:13.575 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:13.440+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:13.575 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:13 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:13.440+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:13.831 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:13 smithi186 podman[70403]: 2024-03-26 05:06:13.486026096 +0000 UTC m=+0.018099679 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:06:13.831 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:13 smithi186 podman[70403]: 2024-03-26 05:06:13.613425962 +0000 UTC m=+0.145499544 container create 0ca8c9bc9d9293562db47d3d9677dbc07f042af89c83cd42632efce79a6e58da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:06:13.831 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:13 smithi186 podman[70403]: 2024-03-26 05:06:13.840299212 +0000 UTC m=+0.372372795 container init 0ca8c9bc9d9293562db47d3d9677dbc07f042af89c83cd42632efce79a6e58da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T05:06:13.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:13 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:13.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:06:13.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:13 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:13.575Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:06:13.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:13 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:13.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:14.136 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:13 smithi186 podman[70403]: 2024-03-26 05:06:13.843517568 +0000 UTC m=+0.375591152 container start 0ca8c9bc9d9293562db47d3d9677dbc07f042af89c83cd42632efce79a6e58da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:06:14.136 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:13 smithi186 podman[70403]: 2024-03-26 05:06:13.92370971 +0000 UTC m=+0.455783296 container attach 0ca8c9bc9d9293562db47d3d9677dbc07f042af89c83cd42632efce79a6e58da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:06:14.426 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 26 05:06:14 smithi186 podman[70403]: 2024-03-26 05:06:14.144743467 +0000 UTC m=+0.676817051 container died 0ca8c9bc9d9293562db47d3d9677dbc07f042af89c83cd42632efce79a6e58da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-03-26T05:06:14.676 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:14 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:14.500+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:14.677 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:14 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:14.500+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:14.677 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:14 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:14.500+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:14.677 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:14 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:14.462+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:14.677 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:14 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:14.462+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:14.677 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:14 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:14.462+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:14.705 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T05:06:14.747 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:06:14.748 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T05:06:14.748 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T05:06:14.748 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.6 2024-03-26T05:06:14.955 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:14 smithi186 systemd[1]: Stopping Ceph osd.6 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:15.411 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:15.070+0000 7f53c0c74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:06:15.412 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:15.070+0000 7f53c0c74700 -1 osd.6 789 *** Got signal Terminated *** 2024-03-26T05:06:15.412 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:15.070+0000 7f53c0c74700 -1 osd.6 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:06:15.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:15.488+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:15.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:15.488+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:15.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:15.488+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:15.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:15.419+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:15.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:15.419+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:15.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:15 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:15.419+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:16.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:16 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:16.454+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:16.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:16 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:16.454+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:16.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:16 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:16.454+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:16.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:16 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:16.433+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:16.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:16 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:16.433+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:16.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:16 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:16.433+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:17.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:17.405+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:17.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:17.405+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:17.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:17.405+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:17.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:17.415+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:17.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:17.415+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:17.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:17.415+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:17.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:17 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:17.415+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:18.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:18.439+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:18.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:18.439+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:18.706 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:18.439+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:18.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:18.427+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:18.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:18.427+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:18.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:18.427+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:18.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:18 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:18.427+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:19.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:19.401+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:28.933359+0000 front 2024-03-26T05:05:28.933430+0000 (oldest deadline 2024-03-26T05:05:53.633320+0000) 2024-03-26T05:06:19.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:19.401+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.334317+0000 front 2024-03-26T05:05:38.334227+0000 (oldest deadline 2024-03-26T05:06:04.134475+0000) 2024-03-26T05:06:19.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6[50415]: 2024-03-26T05:06:19.401+0000 7f53b9260700 -1 osd.6 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:48.136202+0000 front 2024-03-26T05:05:48.136152+0000 (oldest deadline 2024-03-26T05:06:09.136449+0000) 2024-03-26T05:06:19.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:19.390+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:19.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:19.390+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:19.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:19.390+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:19.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:19 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:19.390+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:20.705 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:20 smithi186 podman[70510]: 2024-03-26 05:06:20.591042021 +0000 UTC m=+5.619495230 container died 7ebd3d7e918c2214b4ae44f304aab6a35a33000de158dd583df6942259698a08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:06:20.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:20 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:20.389+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:20.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:20 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:20.389+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:20.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:20 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:20.389+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:20.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:20 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:20.389+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:21.205 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:20 smithi186 podman[70510]: 2024-03-26 05:06:20.963380934 +0000 UTC m=+5.991834142 container cleanup 7ebd3d7e918c2214b4ae44f304aab6a35a33000de158dd583df6942259698a08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T05:06:21.206 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:20 smithi186 bash[70510]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6 2024-03-26T05:06:21.514 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:21 smithi186 podman[70529]: 2024-03-26 05:06:21.214556353 +0000 UTC m=+0.619560734 container remove 7ebd3d7e918c2214b4ae44f304aab6a35a33000de158dd583df6942259698a08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T05:06:21.515 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:21.374+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:21.515 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:21.374+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:21.515 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:21.374+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:21.515 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:21 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:21.374+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:21.842 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:21 smithi186 podman[70617]: 2024-03-26 05:06:21.523743582 +0000 UTC m=+0.110847478 container create fdf3de593ca99cb20f2fd741837e0e7f7308edda52d0b8c47477dac1c9fac996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T05:06:21.843 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:21 smithi186 podman[70617]: 2024-03-26 05:06:21.43041847 +0000 UTC m=+0.017522366 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:06:21.843 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:21 smithi186 podman[70617]: 2024-03-26 05:06:21.751258039 +0000 UTC m=+0.338361939 container init fdf3de593ca99cb20f2fd741837e0e7f7308edda52d0b8c47477dac1c9fac996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T05:06:21.843 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:21 smithi186 podman[70617]: 2024-03-26 05:06:21.754322049 +0000 UTC m=+0.341425946 container start fdf3de593ca99cb20f2fd741837e0e7f7308edda52d0b8c47477dac1c9fac996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:06:21.843 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:21 smithi186 podman[70617]: 2024-03-26 05:06:21.851252019 +0000 UTC m=+0.438355934 container attach fdf3de593ca99cb20f2fd741837e0e7f7308edda52d0b8c47477dac1c9fac996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T05:06:22.205 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 26 05:06:22 smithi186 podman[70617]: 2024-03-26 05:06:22.050523344 +0000 UTC m=+0.637627240 container died fdf3de593ca99cb20f2fd741837e0e7f7308edda52d0b8c47477dac1c9fac996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:06:22.604 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:22.417+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:22.604 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:22.417+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:22.604 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:22.417+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:22.605 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:22 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:22.417+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:22.634 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T05:06:22.676 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:06:22.677 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T05:06:22.677 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T05:06:22.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.7 2024-03-26T05:06:22.955 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:22 smithi186 systemd[1]: Stopping Ceph osd.7 for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.114+0000 7fedf7b18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.114+0000 7fedf7b18700 -1 osd.7 789 *** Got signal Terminated *** 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.114+0000 7fedf7b18700 -1 osd.7 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.398+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.398+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.398+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:23.391 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:23 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:23.398+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:23.826 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:23 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:23.575Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:23.826 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:23 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:23.576Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:06:23.826 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:23 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:23.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:24.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:24 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:24.444+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:24.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:24 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:24.444+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:24.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:24 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:24.444+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:24.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:24 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:24.444+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:25.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:25 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:25.448+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:25.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:25 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:25.448+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:25.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:25 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:25.448+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:25.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:25 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:25.448+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:25.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:25 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:25.448+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.186:6804 osd.4 since back 2024-03-26T05:06:03.134806+0000 front 2024-03-26T05:06:03.134905+0000 (oldest deadline 2024-03-26T05:06:25.434846+0000) 2024-03-26T05:06:26.455 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:26.163Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:06:26.456 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:26.164Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:06:26.456 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:26.164Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:06:26.456 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:26.165Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.12:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.12:8765: connect: connection refused" 2024-03-26T05:06:26.955 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:26.466+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:26.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:26.466+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:26.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:26.466+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:26.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:26.466+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:26.956 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:26 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:26.466+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.186:6804 osd.4 since back 2024-03-26T05:06:03.134806+0000 front 2024-03-26T05:06:03.134905+0000 (oldest deadline 2024-03-26T05:06:25.434846+0000) 2024-03-26T05:06:27.705 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:27 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:27.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-03-26T05:05:29.430051+0000 front 2024-03-26T05:05:29.429988+0000 (oldest deadline 2024-03-26T05:05:54.730041+0000) 2024-03-26T05:06:27.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:27 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:27.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6814 osd.1 since back 2024-03-26T05:05:38.231203+0000 front 2024-03-26T05:05:38.231284+0000 (oldest deadline 2024-03-26T05:06:01.131074+0000) 2024-03-26T05:06:27.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:27 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:27.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6822 osd.2 since back 2024-03-26T05:05:46.832555+0000 front 2024-03-26T05:05:46.832626+0000 (oldest deadline 2024-03-26T05:06:12.033139+0000) 2024-03-26T05:06:27.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:27 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:27.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.12:6830 osd.3 since back 2024-03-26T05:05:52.033627+0000 front 2024-03-26T05:05:52.033607+0000 (oldest deadline 2024-03-26T05:06:16.733613+0000) 2024-03-26T05:06:27.706 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:27 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7[56063]: 2024-03-26T05:06:27.430+0000 7fedf0104700 -1 osd.7 789 heartbeat_check: no reply from 172.21.15.186:6804 osd.4 since back 2024-03-26T05:06:03.134806+0000 front 2024-03-26T05:06:03.134905+0000 (oldest deadline 2024-03-26T05:06:25.434846+0000) 2024-03-26T05:06:28.833 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:28 smithi186 podman[70723]: 2024-03-26 05:06:28.572531741 +0000 UTC m=+5.668589876 container died 93e329742f45c7570908b50ca1f7e616f184c03f457f71c412132ebb594cc446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T05:06:29.205 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:28 smithi186 podman[70723]: 2024-03-26 05:06:28.915243632 +0000 UTC m=+6.011301775 container cleanup 93e329742f45c7570908b50ca1f7e616f184c03f457f71c412132ebb594cc446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T05:06:29.205 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:28 smithi186 bash[70723]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7 2024-03-26T05:06:29.206 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:29 smithi186 podman[70742]: 2024-03-26 05:06:29.074500127 +0000 UTC m=+0.503118250 container remove 93e329742f45c7570908b50ca1f7e616f184c03f457f71c412132ebb594cc446 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True) 2024-03-26T05:06:29.785 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:29 smithi186 podman[70835]: 2024-03-26 05:06:29.385067842 +0000 UTC m=+0.018413732 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T05:06:29.786 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:29 smithi186 podman[70835]: 2024-03-26 05:06:29.500559807 +0000 UTC m=+0.133905698 container create a78e8a7603c9f2f0759e6892843d043a76658fe1bb5292752b0485ad3dbbaf02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T05:06:29.786 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:29 smithi186 podman[70835]: 2024-03-26 05:06:29.711222988 +0000 UTC m=+0.344568878 container init a78e8a7603c9f2f0759e6892843d043a76658fe1bb5292752b0485ad3dbbaf02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:06:29.786 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:29 smithi186 podman[70835]: 2024-03-26 05:06:29.714344739 +0000 UTC m=+0.347690634 container start a78e8a7603c9f2f0759e6892843d043a76658fe1bb5292752b0485ad3dbbaf02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T05:06:29.786 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:29 smithi186 podman[70835]: 2024-03-26 05:06:29.794389282 +0000 UTC m=+0.427735173 container attach a78e8a7603c9f2f0759e6892843d043a76658fe1bb5292752b0485ad3dbbaf02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T05:06:30.205 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:30 smithi186 podman[70835]: 2024-03-26 05:06:30.007467981 +0000 UTC m=+0.640813872 container died a78e8a7603c9f2f0759e6892843d043a76658fe1bb5292752b0485ad3dbbaf02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T05:06:30.547 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 26 05:06:30 smithi186 podman[70861]: 2024-03-26 05:06:30.539290232 +0000 UTC m=+0.526449634 container remove a78e8a7603c9f2f0759e6892843d043a76658fe1bb5292752b0485ad3dbbaf02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T05:06:30.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T05:06:30.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:06:30.595 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T05:06:30.595 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T05:06:30.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T05:06:30.897 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:30 smithi012 systemd[1]: Stopping Ceph rgw.foo.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:31.397 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:30 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a[68112]: 2024-03-26T05:06:30.904+0000 7f5639b5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T05:06:31.397 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:30 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a[68112]: 2024-03-26T05:06:30.904+0000 7f56483dbac0 -1 shutting down 2024-03-26T05:06:33.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:33 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:33.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:33.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:33 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:33.577Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:06:33.897 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:33 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:33.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:41.195 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:40 smithi012 bash[151896]: time="2024-03-26T05:06:40Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T05:06:41.195 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:40 smithi012 podman[151896]: 2024-03-26 05:06:40.936728997 +0000 UTC m=+10.134845662 container died f5bdc9f0bdbe7afbcbd475d3a430bdfcfcffbad5c50bb4fb4eb1d863b19f84cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T05:06:41.520 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:41 smithi012 podman[151896]: 2024-03-26 05:06:41.294738455 +0000 UTC m=+10.492855127 container cleanup f5bdc9f0bdbe7afbcbd475d3a430bdfcfcffbad5c50bb4fb4eb1d863b19f84cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T05:06:41.520 INFO:journalctl@ceph.rgw.foo.a.smithi012.stdout:Mar 26 05:06:41 smithi012 bash[151896]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T05:06:41.644 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T05:06:41.682 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:06:41.682 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T05:06:41.683 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T05:06:41.683 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T05:06:42.041 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:41 smithi186 systemd[1]: Stopping Ceph prometheus.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:42.360 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:06:42.360 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.049Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.050Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.050Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a[65566]: ts=2024-03-26T05:06:42.050Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T05:06:42.361 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 podman[70941]: 2024-03-26 05:06:42.059658949 +0000 UTC m=+0.152972053 container died a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:42.685 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 podman[70941]: 2024-03-26 05:06:42.476386305 +0000 UTC m=+0.569699405 container cleanup a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:42.685 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 bash[70941]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T05:06:42.941 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 26 05:06:42 smithi186 podman[70955]: 2024-03-26 05:06:42.694113121 +0000 UTC m=+0.632982525 container remove a8cd3276e5cecad5812444eeeff593229ce14f882fdffbba2f4702816329b8a3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:42.946 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T05:06:42.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T05:06:42.994 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T05:06:42.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T05:06:43.191 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T05:06:43.857 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:43.577Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:43.857 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:43.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi012.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.12:8443: connect: connection refused" 2024-03-26T05:06:43.857 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:43 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:43.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-26T05:06:44.896 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:44 smithi012 systemd[1]: Stopping Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:45.325 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a[77733]: ts=2024-03-26T05:06:45.032Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T05:06:45.325 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 podman[152392]: 2024-03-26 05:06:45.073780985 +0000 UTC m=+0.134030566 container died 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:45.647 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 podman[152392]: 2024-03-26 05:06:45.432993931 +0000 UTC m=+0.493243520 container cleanup 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:45.647 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 bash[152392]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T05:06:46.099 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 podman[152406]: 2024-03-26 05:06:45.692474272 +0000 UTC m=+0.620831861 container remove 0aa0f27e7ae64c3bfe6cea203a14a56b122a8f1f2ad9bd1086a97ebdaa137d06 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:46.099 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 podman[152406]: 2024-03-26 05:06:45.768016923 +0000 UTC m=+0.696374514 volume remove 60636a367c1d1050175b38790ce272cfccb84f26c90b9115b93c3caa83a574e1 2024-03-26T05:06:46.099 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T05:06:46.099 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 systemd[1]: Stopped Ceph alertmanager.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T05:06:46.099 INFO:journalctl@ceph.alertmanager.a.smithi012.stdout:Mar 26 05:06:45 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.737s CPU time. 2024-03-26T05:06:46.397 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 systemd[1]: Stopping Ceph node-exporter.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:46.397 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 podman[152516]: 2024-03-26 05:06:46.347864518 +0000 UTC m=+0.100086948 container died 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:46.829 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 podman[152516]: 2024-03-26 05:06:46.645446451 +0000 UTC m=+0.397668878 container cleanup 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:46.829 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 bash[152516]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T05:06:47.105 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 podman[152530]: 2024-03-26 05:06:46.838481557 +0000 UTC m=+0.495493935 container remove 76525ad5dd2f2a23a37475c5996ccbd34e9621777a59ec5682afafc99cf38969 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T05:06:47.105 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:06:47.105 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T05:06:47.105 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 systemd[1]: Stopped Ceph node-exporter.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T05:06:47.105 INFO:journalctl@ceph.node-exporter.a.smithi012.stdout:Mar 26 05:06:46 smithi012 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 3.543s CPU time. 2024-03-26T05:06:47.598 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T05:06:47.791 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T05:06:49.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:49 smithi186 systemd[1]: Stopping Ceph iscsi.iscsi.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:06:49.705 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:49 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a[60643]: debug Shutdown received 2024-03-26T05:06:59.298 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:06:59 smithi186 ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a[64220]: logger=cleanup t=2024-03-26T05:06:59.02231533Z level=info msg="Completed cleanup jobs" duration=3.21956ms 2024-03-26T05:06:59.580 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:59 smithi186 bash[71361]: time="2024-03-26T05:06:59Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T05:06:59.580 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:59 smithi186 podman[71361]: 2024-03-26 05:06:59.320492082 +0000 UTC m=+10.121768960 container died 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T05:06:59.947 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:59 smithi186 podman[71361]: 2024-03-26 05:06:59.679765494 +0000 UTC m=+10.481042365 container cleanup 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T05:06:59.947 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:59 smithi186 bash[71361]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T05:07:00.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:59 smithi186 podman[71376]: 2024-03-26 05:06:59.956092666 +0000 UTC m=+0.636889072 container remove 58c86050c6d806314b03d57e2ef05c5eafb90cd81f6b47c3e83c2412899ddfd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-iscsi-iscsi-a, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T05:07:00.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:06:59 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T05:07:00.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:07:00 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T05:07:00.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:07:00 smithi186 systemd[1]: Stopped Ceph iscsi.iscsi.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T05:07:00.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi186.stdout:Mar 26 05:07:00 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 3.476s CPU time. 2024-03-26T05:07:01.705 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:01 smithi186 systemd[1]: Stopping Ceph grafana.a for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:07:02.053 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:01 smithi186 podman[71637]: 2024-03-26 05:07:01.756782453 +0000 UTC m=+0.113959450 container died c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vcs-type=git, version=8.5, description=Ceph Grafana Container, distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vendor=Red Hat, Inc.) 2024-03-26T05:07:02.054 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 podman[71637]: 2024-03-26 05:07:02.062763752 +0000 UTC m=+0.419940746 container cleanup c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, name=ubi8, io.openshift.expose-services=, distribution-scope=public, io.buildah.version=1.32.0, version=8.5, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T05:07:02.361 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 bash[71637]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a 2024-03-26T05:07:02.362 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 podman[71651]: 2024-03-26 05:07:02.272402256 +0000 UTC m=+0.518394358 container remove c74899846e150ad4b40aec74ea58875d16f31cfc7e4513240dab9c243e864ef4 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.openshift.expose-services=, release=239.1651231664, io.buildah.version=1.32.0, vcs-type=git, com.redhat.component=ubi8-container) 2024-03-26T05:07:02.362 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:07:02.705 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:02 smithi186 systemd[1]: Stopping Ceph node-exporter.b for c3b00422-eb2a-11ee-95d0-87774f69a715... 2024-03-26T05:07:02.705 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T05:07:02.705 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 systemd[1]: Stopped Ceph grafana.a for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T05:07:02.705 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 26 05:07:02 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@grafana.a.service: Consumed 11.078s CPU time. 2024-03-26T05:07:03.083 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:02 smithi186 podman[71763]: 2024-03-26 05:07:02.827792386 +0000 UTC m=+0.114479482 container died 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:07:03.408 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 podman[71763]: 2024-03-26 05:07:03.191934569 +0000 UTC m=+0.478621668 container cleanup 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:07:03.409 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 bash[71763]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T05:07:03.688 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 podman[71776]: 2024-03-26 05:07:03.417891917 +0000 UTC m=+0.591247622 container remove 0b7da9301e1a162789630728f8269d024423388e5e3a150dc92eb80b47831c01 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c3b00422-eb2a-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T05:07:03.689 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T05:07:03.689 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T05:07:03.689 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 systemd[1]: Stopped Ceph node-exporter.b for c3b00422-eb2a-11ee-95d0-87774f69a715. 2024-03-26T05:07:03.689 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 26 05:07:03 smithi186 systemd[1]: ceph-c3b00422-eb2a-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 3.475s CPU time. 2024-03-26T05:07:04.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:07:04.193 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T05:07:04.223 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T05:07:04.226 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/remote/smithi012/crash 2024-03-26T05:07:04.227 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/crash -- . 2024-03-26T05:07:04.277 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T05:07:04.277 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now 2024-03-26T05:07:04.278 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/remote/smithi186/crash 2024-03-26T05:07:04.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/crash -- . 2024-03-26T05:07:04.314 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T05:07:04.314 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-03-26T05:07:04.316 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T05:07:04.316 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:07:04.349 INFO:teuthology.orchestra.run.smithi012.stdout:2024-03-26T04:50:00.000191+0000 mon.a (mon.0) 1630 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-26T05:07:04.350 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T05:07:04.350 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[SEC\]' /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:07:04.421 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]' /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:07:04.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[WRN\]' /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T05:07:04.567 INFO:teuthology.orchestra.run.smithi012.stdout:2024-03-26T04:50:00.000191+0000 mon.a (mon.0) 1630 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering 2024-03-26T05:07:04.567 INFO:tasks.cephadm:Compressing logs... 2024-03-26T05:07:04.567 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T05:07:04.610 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T05:07:04.634 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T05:07:04.634 INFO:teuthology.orchestra.run.smithi012.stderr:: No such file or directory 2024-03-26T05:07:04.634 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T05:07:04.634 INFO:teuthology.orchestra.run.smithi012.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T05:07:04.637 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T05:07:04.637 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T05:07:04.638 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T05:07:04.638 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T05:07:04.638 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T05:07:04.639 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T05:07:04.639 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-03-26T05:07:04.639 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T05:07:04.639 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T05:07:04.639 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T05:07:04.639 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.6.log: /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/tcmu-runner.log: 66.7% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr: 85.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T05:07:04.644 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T05:07:04.645 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mgr.y.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T05:07:04.647 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T05:07:04.647 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log: 87.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T05:07:04.648 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.b.log: 92.2% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T05:07:04.650 INFO:teuthology.orchestra.run.smithi186.stderr: 92.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T05:07:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.0.log: 95.2% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T05:07:04.654 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log 2024-03-26T05:07:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T05:07:04.656 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T05:07:04.656 INFO:teuthology.orchestra.run.smithi186.stderr: 91.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T05:07:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log: 92.7% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T05:07:04.667 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T05:07:04.675 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T05:07:04.684 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-volume.log: 94.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T05:07:04.704 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T05:07:05.094 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T05:07:05.589 INFO:teuthology.orchestra.run.smithi186.stderr: 92.9% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T05:07:05.975 INFO:teuthology.orchestra.run.smithi012.stderr: 92.5% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T05:07:07.350 INFO:teuthology.orchestra.run.smithi012.stderr: 92.1% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T05:07:11.172 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T05:07:11.410 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T05:07:11.658 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T05:07:11.793 INFO:teuthology.orchestra.run.smithi012.stderr: 93.8% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T05:07:11.865 INFO:teuthology.orchestra.run.smithi012.stderr: 93.8% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T05:07:12.059 INFO:teuthology.orchestra.run.smithi186.stderr: 93.9% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T05:07:12.061 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-26T05:07:12.061 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m7.436s 2024-03-26T05:07:12.061 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m27.726s 2024-03-26T05:07:12.061 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.983s 2024-03-26T05:07:12.357 INFO:teuthology.orchestra.run.smithi012.stderr: 93.8% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T05:07:12.672 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/c3b00422-eb2a-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T05:07:12.674 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-03-26T05:07:12.674 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m8.053s 2024-03-26T05:07:12.674 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m33.536s 2024-03-26T05:07:12.675 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m1.072s 2024-03-26T05:07:12.675 INFO:tasks.cephadm:Archiving logs... 2024-03-26T05:07:12.675 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/remote/smithi012/log 2024-03-26T05:07:12.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T05:07:17.557 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/remote/smithi186/log 2024-03-26T05:07:17.558 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T05:07:21.915 INFO:tasks.cephadm:Removing cluster... 2024-03-26T05:07:21.915 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 --force 2024-03-26T05:07:22.114 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T05:07:22.409 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c3b00422-eb2a-11ee-95d0-87774f69a715 --force 2024-03-26T05:07:22.606 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: c3b00422-eb2a-11ee-95d0-87774f69a715 2024-03-26T05:07:22.896 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T05:07:22.897 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T05:07:22.912 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T05:07:22.928 INFO:tasks.cephadm:Teardown complete 2024-03-26T05:07:22.928 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T05:07:22.938 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T05:07:22.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T05:07:22.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T05:07:23.048 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T05:07:23.057 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T05:07:23.057 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T05:07:23.059 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T05:07:23.071 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-03-26T05:07:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-26T05:07:23.075 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T05:07:23.076 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-03-26T05:07:23.076 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +655us[ +655us] +/- 74ms 2024-03-26T05:07:23.076 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -686us[ -686us] +/- 72ms 2024-03-26T05:07:23.076 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +53us[ +53us] +/- 75ms 2024-03-26T05:07:23.076 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:07:23.077 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T05:07:23.077 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-26T05:07:23.077 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +656us[ +656us] +/- 74ms 2024-03-26T05:07:23.077 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -674us[ -673us] +/- 72ms 2024-03-26T05:07:23.077 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +49us[ +49us] +/- 75ms 2024-03-26T05:07:23.077 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T05:07:23.078 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T05:07:23.087 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T05:07:23.088 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T05:07:23.102 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T05:07:23.193 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T05:07:23.265 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T05:07:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.295 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T05:07:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:23.326 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T05:07:23.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T05:07:23.352 INFO:teuthology.task.internal:Duration was 2100.594476 seconds 2024-03-26T05:07:23.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T05:07:23.369 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T05:07:23.369 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T05:07:23.372 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T05:07:23.413 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T05:07:23.415 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T05:07:23.747 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T05:07:23.747 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-03-26T05:07:23.747 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T05:07:23.767 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-03-26T05:07:23.769 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T05:07:23.794 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T05:07:23.794 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T05:07:23.810 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T05:07:23.871 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T05:07:23.871 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T05:07:23.878 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T05:07:24.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T05:07:24.311 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T05:07:24.311 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T05:07:24.338 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T05:07:24.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T05:07:24.377 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:07:24.381 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:07:24.403 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-03-26T05:07:24.433 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-03-26T05:07:24.459 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:07:24.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.473 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T05:07:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T05:07:24.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T05:07:24.524 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T05:07:24.525 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/remote/smithi012 2024-03-26T05:07:24.525 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T05:07:24.594 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029/remote/smithi186 2024-03-26T05:07:24.594 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T05:07:24.633 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T05:07:24.633 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T05:07:24.636 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T05:07:24.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T05:07:24.694 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T05:07:24.694 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T05:07:24.804 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T05:07:24.805 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T05:07:24.807 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T05:07:24.819 INFO:teuthology.orchestra.run.smithi012.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 05:07 /home/ubuntu/cephtest 2024-03-26T05:07:24.821 INFO:teuthology.orchestra.run.smithi186.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 05:07 /home/ubuntu/cephtest 2024-03-26T05:07:24.822 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T05:07:24.833 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T05:07:24.870 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T05:07:24.897 DEBUG:teuthology.nuke:shortname: smithi012 2024-03-26T05:07:24.897 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T05:07:24.923 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.726972', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNGiafEtzcNaIpyCQD5qdE71/Oo+/biy6vk9ctDr7hWRANeqskraL43haMAU048ObshHA6lh5GDuWWa8J9fngZg='} 2024-03-26T05:07:24.927 DEBUG:teuthology.nuke:shortname: smithi186 2024-03-26T05:07:24.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T05:07:24.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:22:39.728498', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITubfsMWWTDhcw8ayx3uCTttBhyLSzVmOig8oejOE/sjV+55swSQI7hWPvPptVuaICc2+CZcmlD32vjm4LCcLk='} 2024-03-26T05:07:24.984 INFO:teuthology.orchestra.console.smithi012:Power off 2024-03-26T05:07:24.984 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T05:07:25.011 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-03-26T05:07:25.012 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:07:25.027 INFO:teuthology.orchestra.console.smithi186:Power off 2024-03-26T05:07:25.027 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T05:07:25.036 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-26T05:07:25.050 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-03-26T05:07:25.050 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:07:25.074 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-26T05:07:29.038 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:07:29.075 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:07:29.148 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-03-26T05:07:29.185 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-26T05:07:33.150 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:07:33.186 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T05:07:33.262 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-03-26T05:07:33.297 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-03-26T05:07:33.363 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-03-26T05:07:33.398 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-03-26T05:07:33.502 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} duration: 2100.5944764614105 failure_reason: '"2024-03-26T04:50:00.000191+0000 mon.a (mon.0) 1630 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-26T05:07:33.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T05:07:33.578 INFO:teuthology.run:FAIL